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EXECUTIVE SUMMARY 
Introduction 


This evaluation of the Sudburj- District Nurse Practitioner Clinics (SDNPCs) was conducted by 
PRA Inc. for Ontario’s Ministry of Health and Long-Tenn Care (MoHLTC). The purpose of the 
study is to identify lessons learned at the SDNPC, which will help to inform the roll-out of the 
subsequent Nurse Practitioner-Led Clinics in Ontario. 

Program Description 

In November, 2007, the Ontario government announced that 25 Nurse Practitioner-Led Clinics 
would be established in the province. The establishment of these clinics is intended to reduce the 
number of people without primary health care providers and aims to improve the 
comprehensiveness and integration of services. Nurse Practitioner-Led Clinics will be composed 
of a number of health care professionals including at a minimum Nurse Practitioners (NPs) and 
physicians. 

SDNPCs opened in August of 2007 and sen’e as a pilot project for the initiative. The Ministry 
provides funding for six full-time NP positions and the clinic has two part-time con-sulting 
physicians who see patients and provide formal and informal consultation to the NPs. The 
physicians receive a monthly stipend for the consultations they provide and are compensated 
with fce-for-scrvices (FFS) for seeing patients in appointments. Patients of the SDNPC are 
assigned to a specific NP as their primary health care provider; however, patients are registered 
with the clinic itself. The Ministry expects 4,800 patients to be registered with the clinic after the 
clinic’s first three years of operation {i.e., by August 2010). 

Methodology 

Tn consultation with the Ministry, PRA developed a list of evaluation questions to guide this 
study (Table 4 in this report), Tlie questions were designed to align with the guidelines set out in 
the RFP. They cover the issues of rationale, design and delivery, satisfaction and impact. 

We used the following lines of evidence in this evaluation: 

Document review 

- Key informant interviews (n = 19 interviewees) 

- Patient focus groups (n = 20 participants) 

- Patient feedback survey (n = 603 respondents) 


Findings 

The findings are presented below in five general themes that emerged from this study: awareness 
and understanding, implementation, physician integration, the clinic model, and patient 
satisfaction. 
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Awareness and understanding. There appears to be a high level of public awareness about the 
existence of the SDNPC. In addition, patients of the clinic appear to understand the clinic model 
and the role of an NP. However, the understanding of the model and the NP role is questionable 
with regard to the general public. 

Most participants in the focus groups came to understand the role of an NP after their first 
appointment, when their NP explained their role and the differences between an NP and a 
physician. A total of 91% of survey respondents believe that they have a clear understanding of 
the differences betw^een an NP and a physician. 

Focus group participants and key informants perceived there to be a high level of awareness 
about the existence of the clinic in Sudbury. The document review^ revealed that the clinic has 
received a substantia] amount of media attention, both positive and negative, that has likely 
contributed to the apparent high level of awareness. The negative media attention was generated 
by physicians who questioned the effectiveness, quality of care received, and potential costs of 
the clinics. This may have created a lack of confidence in the clinic and initiative, and an 
impression that there is dissention among health care workers. According to focus group 
participants, the negative press has caused some individuals to question the clinic and more 
generally this model of care. 

Implementation, Implementation issues arising fi:om this study were: concerns with the 
Electronic Management Records (EMR) system, delays in receiving budgetary approvals and 
funding, space, and patient complexity'. The EMR did not function smoothly for the fir-st year and 
a half of the clinic’s operations and there was a lack of assistance from the selected vendor. The 
clinic has since noticed improvements and clinic NPs highlighted the benefits of having an EMR 
system when multiple professionals are providing care to the same individual. 

The clinic received good project management support from of the Ministry'. However, clinic 
management said that delays in receiving budgelao' approvals and funding from the Ministry' 
have impacted clinic operations. In addition, the clinic management. Board members and NPs 
said that a larger facility is needed for the clinic. The opening of the next clinic in Lively will 
apparently not alleviate this concern. This issue, to some extent, limits the number of patients 
that can be registered at the clinic. However, a more important factor limiting the number of 
patients that can be registered with the Ri\'erside clinic has been patient complexity. 

New patients of the clinic may not have received health care in a number of years; therefore, 
patients of the clinic can have complex medical needs. Medically complex patients require more 
attention and more physician involvement, which affects the number of patients that can be 
registered with the clinic. The Ministry expects a patient registry of 4,800 by August, 2010; 
however, it is unlikely that the clinic will meet this target. 

Physician integration. The current physician compensation model is not appropriate, and key 
informants across all stakeholder groups recommended increased physician compensation. 
Physicians see patients with the most complex health needs who require longer visits. This has 
fin^cial implications because it translates to seeing fewer patients per day and fewer billable 
visits. Additionally, the current compensation model does not promote physician involvement in 
the clinic. 





Ministry of Health and Long~Term Care 3 

EvaluatUm of the Sudbury District Nurse Practitioner Clhties—October 1, 2009 

A number of key informants said that a salary model for physicians of the NP-Led Clinics could 
help overcome the barriers to collaboration, as long as the salary is sufficient. Some key 
informants said that having FFS compensation in place at the clinic is not ideal because it 
promotes different views of time management when physicians are the only team members not 
on salary. How'ever, the Ministry has a ftmding agreement with the Ontario Medical Association 
(OMA) whereby OMA approval is required for any changes to the current funding model. 

NP-Led Clinic model. The clinic model appears to work well, with the exception of the 
aforementioned issue of physician compensation and the related issue of limited physician 
involvement in the clinic. The system of patient registration to the clinic as opposed to enrolment 
with an individual health care provider w'as perceived as a benefit of the NP-Led Clinic model by 
patients, clinic staff clinic management and physicians because it helps ensure continued access 
to care for indiwduals registered with the clinic. In addition, the clinic model seems to facilitate 
NPs in functioning to their full scope of practice. However, there w'as a learning curve in terms 
of how' the roles of different team members are played in the run of a day. 

Key informants were generally satisfied with the system of accountability within the clinic. Most 
key informants supported having the Clinic Director be an NP, because knowdedge of clinical 
issues was seen as essential to management decisions. However, several key informants said that 
the Clinic Director in future clinics does not necessarily need to be an NP, although it would be 
necessaiy to have a senior NP or physician in charge of the main clinical decisions. 

The clinic completes reporting forms to track patient encounters and to track physician 
consulting and sends the completed forms to the Ministry. The usefulness of both of these 
reporting systems is questionable. NPs and clinic management said that the reporting system is 
not capturing appropriate results. In addition, a number of key informants across different 
stakeholder groups are not aware of how this information is being used. 

Patient satisfaction. Patients showed an overwhelmingly high level of satisfaction with the 
services they receive from their NP. The main reasons for high satisfaction were the attitudes of 
the NPs and the thoroughness of care provided. When compared with their previous health care 
situations, patients’ wait times have decreased. In addition, patients arc pleased with the health 
education they receive from their NP, Since the clinic opened in Sudbury, the clinic’s patients 
believe that their access to health care has improved. 

Conclusions and lessons learned 

Conclusions and lessons learned are presented under the evaluation issues of rationale, design 
and delivery, satisfaction, and impact. 

Rationale. There were NPs living in Sudbury who were not employed and who lobbied for the 
clinic, which led to its establishment. The clinic was established in Sudbury due to the shortage 
of physicians in the area and the resulting high number of patients with no primary health care 
provider. 

Design and delivery. The current physician compensation model is not appropriate, and w'as 
identified as the most serious concern in this evaluation. If the model is not changed, there will 
be considerable challenges in implementing the next 25 NP-Led Clinics because it will be 
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difficult to recruit and retain physicians. The current model does not account for the medical 
complexity of patients being seen and limits physician involvement and input into the clinic. 
Physicians should become more involved in the clinic which would be possible with a new 
compensation model. If the next 25 NP-Led Clinics are to succeed, negotiations between the 
Ministry and the OMA are necessary in order to establish a compensation model that is attractive 
for physicians. 

With respect to administrative issues at the clinic: The EMR system was seen as essential to the 
operation of the SDNPC, and having an EMR system is recommended for future NP climes. 
Fumre NP-Led Clinics should ensure the selection of a high-quality EMR service provider that is 
available for communication when issues arise. In addition it would be beneficial for all future 
NP-Led Clinics to have in place appropriate clerical support from the onset so that NPs do not 
need to take on clerical duties. This has impacted operations initially in the case of the SDNPC. 
Future clinics should also secure facilities with appropriate square footages from the onset of 
operations, ensuring that the facilities the)' select will meet their immediate and future needs. 
Also, the patient reporting forms and the physician consultation reporting forms need revisions 
so that the information collected is more pertinent. The forms should be re-designed with a goal 
in mind of how the completed forms will be used. 

The management structure with a Clinical Director as an NP appears to work w'ell for the 
SDNPC, but may not be necessary for all future NP-Led Clinics. However, for all future clinics, 
there should be a regulated health professional in charge of all clinical operations to ensure that 
someone with appropriate expertise is involved in clinical decisions made. In addition, for future 
clinics, logistics of the way in which clinic team members play their roles should be discussed at 
the onset of clinic operations. 

NPs at the clinic are functioning to their full scope of practice. The nature of the clinic model 
facilitates this because NPs are primary health care providers and therefore deal with a broad 
range of health issues. The collaborative relationship among NPs and between the NPs and 
physicians is another factor facilitating NPs in functioning to their full scope. Future NP-Lcd 
Clinics could ensure that individuals hired to work at the clinics have the ability to work in a 
team setting. Moreover, physicians could be provided with training about the role of NPs at the 
clinic. 

There is an apparent need to increase public understanding about the role of an NP and the NP- 
Led Clinics initiative in general. The negative media attention received in Sudbury may occur in 
other areas where NP-Led Clinics are being implemented, A provincial media campaign led by 
the Ministry w'as recommended to increase public understanding about the initiative. The 
campaign could focus on educating the public about how' health services arc delivered in this 
type of model, including the roles of NPs and physicians at the clinics. 

The clinic is not likely to reach the target of 4,800 patients by August 2010 while maintaining the 
quality of services it is currently providing. The medical complexity of patients at the clinic is 
affecting the number of patients that can be registered. The Ministry can look to the capacity of 
the SDNPC when planning for future clinics to estimate the number of patients that can be 
served. 
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Satisfaction. Patients are generally highly satisfied with the services they receive from the clinic 
and with the clinic itself, They appreciate the positive and respectful attitude of their NP, and are 
comfortable asking questions. Patients also appreciated having access to physicians when 
needed, and are satislled with the thorough care they receive. They appreciate the amount of time 
that is spent with them per visit and the good listening skills of their NP. 

The NPs, doctors, and other stakeholders who were interviewed generally showed a high level of 
satisfaction with the clinic. NPs appreciate being able to function to their full scope efficiently 
and in an empow^ering setting. 

Impact. Patients registered at the clinic perceive major differences in the primary health care 
they receive since the clinic was opened, especially given that prior to the clinic, they had no 
primary health care provider. By providing primary health care access to individuals who 
previously had none, the clinic appears to be an effective way to improve access to primary 
health care in Sudbury. 

More generally, the next 25 clinics can learn from the experiences of the SDNPC by 
communicating with the SDNPC about implementation challenges that they have faced. The 
Ministry could provide support for the SDNPC to share information about best practices and 
lessons learned with other clinics. In addition, the Ministry^ would need to ensure enough staffing 
at the provincial level to assist with the rollout of the next 25 clinics. 
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1.0 Introduction 

This document constitutes the final report for the Evaluation of the Sudbury District Nurse 
Practitioner Clinics fSDNPCs). This evaluation was conducted by PRA Inc. for Ontario's 
Ministry' of Health and Long-Term Care (MoHLTC, also referred to throughout this report as 
“the Ministry”). 

1.1 Scope and objectives of the evaluation 

The purpose of this evaluation is to identify lessons learned at the SDNPC, which will help to 
inform the roll-out of the subsequent Nurse Practitioner-Led Clinics in Ontario. The main 
evaluation issues we examined were rationale, design and delivery, satisfaction, and impact. 
Evaluation questions under each of these issues were developed and can be found in Table 4 on 
page 11. 

As per the Request for Proposals (RFP), the evaluation examined the following areas: 

► Barriers and facilitator to the implementation of the SDNPC, including stakeholders’ 
perspectives and lessons learned; 

► Patient/client access to comprehensive primary health care services; 

► Patient^client satisfaction; and 

► Integration and collaboration between Nurse Practitioners (NPs), family physicians and 
other providers. 


1.2 Structure of the report 

The report is divided into several sections. Section 2 includes the policy context relating to the 
clinic’s implementation and presents a profile of the clinic. Section 3 describes the methodology 
used to complete the evaluation. In Section 4, the findings from all data collection activities 
carried out as part of the research are presented. Section 5 concludes this report, and discusses 
the lessons learned as they relate to the implementation of the next 25 NP-Led clinics in Ontario. 
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2.0 Program description 

This section discusses the policy context in which the SDNPCs were implemented, followed by a 
profile of the clinic. 

2.1 Policy context 

NPs are Registered Nurses who have “extended class” designations with the College of Nurses 
of Ontario. They have an expanded scope of practice in the areas of health assessment, diagnosis, 
ordering tests, prescribing treatments and health promotion. When patient care needs extend 
beyond the scope of an NP, they consult with a physician in accordance with guidelines 
established by the College of Nurses of Ontario (2009)'. 

NPs have come to be recognized as playing a key role in the health care system. In Ontario, the 
MoMLTC has funded a number of initiatives since 1998 dedicated to creating and sustaining NP 
positions in Ontario, with a focus on underserviced areas (e.g.. Nurse Practitioner Demonstration 
project and funding to increase clinical education positions). The MoHLTC continues to fund 
initiatives involving NPs, most recently with the large-scale NP-Led Clinics initiative. 

In November of 2007, the provincial government announced that 25 NP-Led Clinics would be 
established in Ontario. The establishment of these clinics is intended to reduce the number of 
people without primary health care providers and increase access to health care. The clinics are 
also intended to improve the comprehensiveness and integration of services. The implementation 
of these clinics is part of the Ontario Government’s family Care for All Strategy, which aims to 
provide comprehensive, accessible and timely health care for Ontarians (Government of Ontario, 
2009)^ 

The clinics emphasize chronic disease management and health promotion. This is done through 
treatment and monitoring, as w'ell as through improving the skills of patients to manage their 
own health (Government of Ontario, 2009)'', NP-Led Clinics will be composed of a number of 
health care professionals, including at a minimum NPs and physicians. However, having a range 
of health care professionals in NP clinics is possible, with providers working collaboratively as a 
health care delivery team. The Romanow report (2002)^ determined that this type of inter¬ 
professional care is a direction for the future of health care in Canada. 

The Ministry funds a similar initiative — the Family Health Team (FIdT) Initiative. Like the NP- 
Led Clinics, FHTs aim to provide collaborative health care for individuals who do not have a 
health care provider, A total of 150 FHTs have been created since 2005, and another 50 will be 
implemented, A key difference between FHTs and NP-Led clinics is that with NP-Led clinics, an 
NP acts as the primary provider of health care and serves as a point of entry to the health care 
system. NPs consult with physicians as appropriate. In the FHT model, physicians are the 
primaiy' health care providers. Both the FHTs and the NP-Led Clinics are being implemented 
and planned in regions across the province. 

Selection of sites. The Ministry undertook a needs assessment considering a range of key 
population and health indicators to determine which Local Health Integration Networks (LHINs) 
have the greatest need for additional resources. These LHINs are: North West, North East, Erie 
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St. Clair, North Simcoe Muskoka, Central West, Central East, Champlain, and South East. Key 
indicators for assessing need included: 

- Proportion of unattached patients, excluding the Health Care Connect program; 

- Prevalence of one or more of nine chronic diseases, including diabetes; 

Number of full-time equivalent (FTE) general practitioners / family physicians in the 
LHIN per 10,000 population; and 

- Number of existing FHTs I Community Health Centres. 

To date there have been two calls for applications (waves) for the NP clinics. Three NP-Led 
Clinics W'ere announced in February 2009 as part of Wave 1. It is expected that successful 
applicants for Wave 2 will be announced in the fall of 2009. 

The SDNPC, the subject of this evaluation, was the first established. The SDNPC serves as a 
pilot project for the initiative, and is discussed in the following section. 

2.2 Clinic profile 

Ihis section provides a profile of the SDNPC by discussing the rationale for the pilot project in 
Sudbury leading to the establishment of the clinic. This is follow'ed by a description of the 
clinic’s main activities, management and organization, and resources. 

2.2.1 Rationale and clinic establishment 

Announced in November, 2006, The SDNPC w^as intended as a three-year demonstration project. 
The project was established in Sudbury, Ontario, for several reasons. 

Although Sudbury is the largest city in northern Ontario, it is considered underserviced when it 
comes to health care. On some of the key determinants of health, Sudbury scores lower than the 
rest of the province (e.g., education, income, health (Statistics Canada, 2006)’. Prior to the 
establishment of the clinic, there were a number of NPs in the Sudbury area who were unable to 
tind employment and were available for work. Some were considering moving from Sudbuiy to 
other regions of Ontario or to the United States for employment. 

A group of these NPs collaborated and lobbied for a NP-Led Clinic. They were not successful in 
their application under the FHT Initiative; how’ever, the Ministry^ approved the clinic separately 
in November of 2006, This was one year prior to the aimoiincement of the larger initiative. Funds 
from the Ministry began to flow' in May of 2007. The clinic opened its doors in August of 2007 
and began registering patients immediately. A timeline of the events leading to the establishment 
of the SDNPC is presented below in Table 1. 


Table 1; Timeline of events leading to clinic establishment 

Event 

Month, Year 

SDNPC announced 

November, 2006 

SDNPC received funding from Ministry 

May, 2007 

SDNPC opened 

August, 2007 

NP-Led Clinics Initiative announced 

November, 2007 
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The clinic is located on 359 Riverside Drive in downtown Sudbury. The clinic also serves the 
remote community of Chapleau one day per week (staffed by one NP and one volunteer 
Registered Nurse). Plans are in place for the SDNPC to open another clinic in Lively, which is 
also in the Greater Sudbury' area. The Lively clinic will have two full-time NPs, and the Clinic 
Director’s time will be split between the two sites. The Lively clinic will be under the same 
management as the clinic in Riverside and its opening is planned for January, 2010. This 
evaluation focuses on the Riverside Drive location. 


2.2.2 Clinic activities and model 

'fhe Ministry provides funding for six full-time NP positions at the SDNPC; however, the clinic 
only has a total of 5,5 FTE NPs. The Clinic Director is an NP who spends half of the time as 
Clinic Director and the other half of the time as an NP. The clinic also has two consulting 
physicians. Together, the physicians spend a total of 3.5 days per week on-site (one physician 
spends two full days per week at the clinic and the other spends three halfdays per w-eek). One 
full-time Office Manager oversees administrative aspects of the clinic, and three full-time staff 
members provide clerical support. The clinic has a part-time pharmacist as well. With opening of 
the clinic in Lively, a dietician, social worker and RN will be added to the team. 

Patients with the clinic are assigned to have a specific NP as tlieir primary health care provider. 
However, patients are not “enrolled” with specific individuals; rather, they are registered with 
the clinic in general. In other words, if an NP were to leave the clinic for some reason, patients 
would still be registered with the clinic itself and would be assigned to another NP as their 
primary health care provider. 

Physicians at the clinic can see patients directly when patients are referred by their NP. 

Physicians also provide NPs with consultation, both formally and informally. Formal 
consultation involves set times at the beginning of the day when NPs have the opportunity to ask 
physicians questions. Informal consultations can take place on-site in the form of questions being 
asked throughout the day, or off-site by phone or email. The clinic’s physicians are compensated 
in two ways. They bill fee-for-service (FFS) for patients they see directly, and they receive a 
fixed monthly stipend for the consultation they provide. 

The Ministry' e.xpects 4,800 individuals to be registered with the clinic within the first three years 
of operation (i.e., by August 2010). This is equivalent to 800 patients per NP (since the clinic 
receives funding for six NPs). Internal documents of the Ministry state that the clinic aimed to 
have 2,500 patients registered by the end of the 2008/2009 fiscal year. As of July, 2009, 2,430 
patients were registered. For tracking purposes, patients provide their Ontario Health Insurance 
Plan (OHIP) numbers to the Ministry as part of the registration process. The clinic submits these 
numbers to the Ministry on a monthly basis (personal communication, clinic staff). 
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Table 3 shows numbers of patients registered with the clinic over time. 


Table 2: Number of patients registered at clinic over time 

Date 

Number of patients 
registered at clinic 

August, 2007 

Recifstration began 

April, 2008 

1514 

November, 2008 

1855 

July, 2009 

2430 

Source: Government of Ontario, 2008^ 


A number of programs are available for patients of the clinic. To date, the clinic has offered three 
programs, including a smoking cessation program, a weight loss program and an ITPV 
immunization program. The clinic also provides training opportunities for NPs and medical 
students. 


2.2.3 Management and organization 

NPs at the clinic are accountable to the Clinic Director, w'ho is responsible for the day-to-day 
operations of the clinics. Clerical staff members are accountable to the Office Manager. The 
Clinic Director reports to the Board of Directors (BOD) of the clinic. The Board is responsible 
for establishing bylaws and policies, lobbying for support and ensuring appropriate uses of 
funding. The BOD holds Annual General Meetings and monthly meetings that are attended by 
the Clinic Director and Office Manager. Clinic policy requires at least 50% of Board members to 
be NPs. Each month, the Clinic Director submits a report to the BOD for review^ prior to the 
meeting. The following organizational chart summarizes the management structure at the clinic. 
Arrows indicate lines of accountability. 

Management structure of the SDNPC 



i-— 

Board of Directors 


J 

1 . 



Clinic [ 

}iredor 

- - 


Physicians 


Nurse Practitioners 


Office Manager 

-j- 


Clerical staff 
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2.2.4 Clinic resources 


Table 3: Clinic expenditures 


July 1, 2007 - June 30, 2008 
(12 month period) 

July 1, 2008 - March 31,2009 
(9 month period) 

Salaries 

5613,760 

$663,822 

Operating costs 

$101,367 

$125,987 

Amounts were provided by the SDNPC 


As evident from the table above, the clinic’s budget years were initially not in line with the 
typical fiscal years due to the timing of implementation. In addition, approximately $15,000 of 
funding included under "salaries” (e.g., consulting fees) in the first budget period (July L 2007 - 
June 30, 2008) was moved to “operating costs” in the second budget period (July 1 2008 - 
March 31,2009). 
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3.0 Methodology 

This section of the report provides the evaluation questions and describes the methodology used 
to complete the evaluation. 


3.1 Evaluation questions 

In consultation with the Ministiy, PRA developed a list of evaluation questions to guide this 
study {Table 4). TTie questions were designed to align with the guidelines set out in the RFP. 
They cover the main issues of rationale, design and delivery, satisfaction and impact. Each 
question is responded to individually in Section 5 of this report, under conclusions and lessons 
learned. 


Table 4: Evaluation issues and questions 

Rationale 

1. What considerations led the MoHLTC to develop the Sudbury District NP Clinic? What issue{s) 
was the clinic intended to address? 

Design and delivery 

2. Has the clinic faced challenges in ite implementation? If yes, what were the challenges and how 
were they overcome? 

3. Are the current program delivery mechanisms and structure appropriate and effective? 

4. Does the current reporting system allow for the clinic to adequately communicate their results? 

5. What, if any, issues are there with NP/doctor integration? Is the level of physician consultation 
appropriate? 

6. Are the NPs functioning to the fuli scope of their practice? \A^at factors facilitate or impede the 
NPs from functioning to their full scope? 

7. Are there any concerns about NP role clarity and scope of practice? If so, how could these 
concerns be mitigated? 

8. Is the NP workload appropriate? What factors affect their workload? 

9. Is the current physician compensation model appropriate? Why or why not? 

Satisfyetion 

10. How satisfied are patients with the services provided by the NPs? How satisfied are the patients 
with the clinic itself? 

11. How satisfied are the NPs, doctors, and other stakeholders with the clinic? 

Impact 

12. Do patients perceive any differences in primary health care in their communities since the 
implementation of the Sudbury District NP Clinic? 

13. What impact has the clinic had on access to primary health care? 
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3.2 Document review 

The Ministry and the clinic provided PRA with relevant documentation for review. This included 
internal summary documents, project status updates, the clinic's draft business plan, position 
papers, and articles published in the media regarding the clinic. PRA identified additional 
sources of information throughout the study. This review provided background and contextual 
information for the evaluation, and responds directly to some of the que.stions identified tn the 
evaluation framework. The document re\uew also included a review of a client satisfaction 
survey the clinic conducted internally (SDNPC, 2008a)^ Information from this line of evidence 
has been incorporated into this document (largely in Section 2) and throughout the report. 

Information was also collected from individuals in the following marmer: 

- key infonnatit interviews (Section 3.3); n = 19 interviewees 

- focus groups (Section 3.4); n = 20 participants 

- patient feedback survey (Section 3.5); n = 603 respondents 

3.3 Key informant interviews 

PRA interview'ed a total of 19 key informants. The Ministry identified these individuals in 
consultation with PRA. Key informants included clinic physicians and NPs, clinic management, 
board members, representatives of the MoHLTC, and external stakeholders who had some level 
of involvement or experience with the clinic. To en.sure that the scope of this evaluation 
remained specific to the SDNPC, only individuals with at least some level of direct experience or 
involvement with the clinic were interviewed. The interviews were conducted by telephone or in- 
person, and the findings have been integrated into tliis report. Interview guides were tailored for 
the various key informant groups and are found in Appendix A. 

3.4 Focus groups with patients 

PRA conducted the focus groups with patients in Sudbury, Ontario. Initially, four focu.s groups 
were planned, with one planned for participants who prefer to participate in French. However, no 
patients signed up for the French session. ITierefore, PRA conducted three sessions, and all were 
in English. 

Since PRA did not have access to the names and contact information of clinic patients for 
privacy reasons, alternative recruitment methods were necessary, French and English signs were 
posted in the clinic to advertise the focus groups. Approximately half of the patients registered 
for the groups in this way. To recruit additional participants, clinic staff was made aware of the 
focus groups and mentioned that they would be taking place to patients who happened to have 
appointments during the recruitment period. Finally, focus group recruitment letters were sent to 
100 patients informing them of the sessions and inviting them to participate. 

The focus groups took place during the evenings of May 25 and 26, 2009, with nine, six, and five 
participants in each group. Each participant received $45 for their participation and refreshments 
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were provided. Each focus group was approximately an hour and a half in length. The guide used 
to facilitate the focus groups can be found in Appendix B. 

3.5 Survey of patients 

The purpose of the Patient Feedback Survey was to gather detailed information about the 
satisfaction of the clinic’s patients. Survey packages were distributed to 970 patients, who were 
randomly selected as potential respondents from a list. The sample of 970 represents 
approximately 40% of the clime’s total patients. 

The survey process included the initial mailing of the sutv'ey package and two sets of reminder 
postcards These steps are described in more detail below, 

► First survey mailing. The first mailing included a cover letter in English and French 
(Appendix C), French and English copies of the questionnaire (Appendix D), and a 
postage-paid return envelope. This package of materials was sent to each of the patients 
included in the sample on June 12, 2009. The cover letter requested that patients 
complete and mail their questionnaires by July 17, 2009. 

► Reminder postcards. The first set of reminder postcards was sent approximately ten days 
after the first mailing, and the second reminder followed two weeks later (Appendix E). 

A number of people requested replacement questionnaires because they had accidentally 
misplaced or discarded the original copy. PRA provided these individuals with additional 
questionnaires. 

Table 5 show's important dates associated with the survey 


Table 5: Important survey dates 

Task 

Dates 

! First survey mailing 

June 12, 2009 

First reminder mailing 

June 22, 2009 

Second reminder mailing 

July 7, 2009 

Completion requested by 

July 17, 2D09 

Mail survey cut-off 

July 24, 2009 


► Analysis. Of the 970 questionnaires mailed out, PRA received 603 responses (response 
rale = 62%). This represents a total error margin of ± 4.1%. The data was entered into 
CATI (Computer Assisted Telephone Interviewing) and then pulled into SPSS (the 
Statistical Package for Social Sciences) for cleaning/analysis. Open-ended responses 
were grouped into themes and coded. 

The youngest respondent w'as 16, and the oldest was 93. The survey was provided to all patients 
in both English and Frenchi however, 97% of all surveys were completed in English and only 
3% were completed in French. A total of 64% of respondents were female, 33% were male, and 
3% did not specify their gender. 
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4.0 Key findings 

This section presents the key findings from the document review, interviews, focus groups and 
patient sur\'ey. The section is organized according to the prominent themes that emerged from 
^e study. Findings are grouped in the following five categories: awareness and understanding, 
implementation, physician integration, the clinic model and patient satisfaction. These categories 
are presented in an order that aligns chronologically with the process of establishment of the 
clinic. 


4.1 Awareness and understanding 

There appears to be a high level of public awareness about the existence of the SDNPC. In 
addition, the clinic’s patients appear to understand the clinic model and the general differences 
between the roles of the clinic's NPs and physicians. However, in this area, there appears to be a 
need to increase the understanding of the general public. Awareness and understanding are 
discussed in the subsections below. 


4.1.1 Awareness 

Focus group participants, NPs, Board members, and other stakeholders perceived a high level of 
awareness about the existence of the clinic in Sudbury, This aw'areness was generated in a 
number of ways. The surv'ey and focus groups revealed that patients most frequently learned 
about the clinic through word of mouth, fbllow'ed by community newsletters, or the newspaper. 
Other common ways patients became aware of the clinic were: referrals from other health 
professionals, posters, public intbrmatioti booths, and the radio. The following tabic summarizes 
the most common methods by which patients first heard about the clinic according to the survey. 


Table 6: How did vou first find out about this health clinic? iQ 1.1 

Mode of awareness 

% of patients 
(n=603) 

Friend or family member / word of mouth 

42% 

Community newsletter / newspaper 

29% 

Advertisement 

10% 

Another health professional referred me 

10% 

Public information or display booth 

5% 

Radio 

3% 

Television 

2% 

Community medical clinic 

2% 

[ Other 

4% 

[ Respondents could provide more than one answer; therefore totals sum to more than 100%. 


Clinic management noted that there was little need to advertise the clinic due to high patient 
interest. The high proportion (42%) of survey respondents who learned about the clinic through 
word of mouth supports the idea that little advertisement w^as required. Focus group participants 
and key informants commented anecdotally that there was a rush for people to apply to the clinic 
quickly to help ensure their acceptance. Focus group participants and key informants who were 
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present at the clinic on its opening day said that there was a long line of people waiting to sign 
up. 

The document review revealed that the clinic has received a substantial amount of media 
attention, which has likely contributed to the apparent high level public awareness. ITie attention 
the clinic received was both positive and negative. The positive attention included informational 
pieces about the clinic and letters to the editor from patients supporting the clinic. The clinic’s 
website includes a section entitled "testimonials” where numerous patients commend the clinic 
(SDNPC, 2008b)^. 

The negative media attention was generated by physicians in letters to the editor and published 
articles that questioned the effectiveness, quality of care, and potential costs of the clinics (e.g.. 
The Sudbury Star, 2009*; Northern Life, 2009^). Letters to the editor argued that the government 
should not fund the initiative because the model has yet to be proven (e.g., The Sudbury Star, 
2009 ; Northern Life, 2009^). This may have created a lack of confidence in the clinic and 
initiative, and an impression that there is dissention among health care workers. According to 
focus group participants, the negative press has caused some individuals to question the clinic 
and more generally this model of care. Anecdotal reports suggest that one physician left the 
clinic because of pressure. 

A few focus group participants said either they themselves, or other community members they 
know, were initially critical of the clinic. This feeling was generated at least in part by the 
negative press discussed above. However, focus group participants said that any doubt they were 
experiencing about the clinic disappeared after their first encounter w^ith the clinic. 

4.1.1 Understanding 

Patients of the clinic appear to understand the clinic model and the general role of NPs and 
physicians within the model. However, there is an impression that there is a need to increase the 
understanding of the general public in this area. 

Although several focus group participants did not understand how' NPs are trained, all focus 
group participants were able to correctly identify basic differences between the role of an NP and 
a physician. Participants understood that NPs can prescribe certain medications and order certain 
tests, and that NPs consult with physicians as needed. This finding was apparent in the patient 
survey as well. A total of 91% of survey respondents believe that they have a clear understanding 
of the differences between an NP and a physician. 

Some focus group participants were familiar with the role of an NP prior to registering at the 
clinic. This was because these individuals had previous experience with NPs, knowledge of the 
health care system because of working in the field, or because they learned about NPs and the 
clinic through the media. Some participants in the focus groups came to understand the role of an 
NP only after their first appointment, when their NP explained their role and the differences 
betw'ccn an NP and a physician. A total of 86% of surv'ey respondents said that their NP 
explained their role (Table 7). 
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Table 7: Has anyone ever explained the role of the Nurse Practitioner you are 
seeing at the clinic? (Q. 5) 


% patients 
fns603) 

Yes. the Nurse Practitioner explained it 

86% 

Yes, a physician explamed it 

2% 

No one explained it 

5% 

Other 

4% 

Don't know 

6% 

No response 

1% 

Respondents coufd provide more than one answer: therefore totals sum to more than 100% 


Although the clinic’s patients appear to clearly understand the role of an NP, this is not 
necessarily the case with the general public. As mentioned, many focus group participants were 
not aware of the role of an NP until visiting the clinic, and negative press caused some patients 
and apparently some other community members to be critical of the clinic. Clinic management 
and the BOD devoted a substantial amount of time to dealing with media-related issues. The 
Ministry, the Nurse Practitioners’ Association of Ontario, and the Registered Nurses’ 

Association of Ontario were apparently initially not active in publishing material about the clinic 
in the media. Some key informants believed that if these groups initially expressed support for 
the clinic in the media, the clinic’s image would have been strengthened. The clinic’s Board 
members, NPs, management, and other stakeholders warned that negative media attention may 
occur in other communities where the clinics are being implemented. 

Several key informants perceived the name “NP-Led Climes” as somewhat of a misnomer. 
According to them, the name does not accurately reflect physician involvement, and could lead 
to a misunderstanding that physicians are not involved in the clinics. Although NPs are the 
primary point of entry for patients, key informants did not always believe that the clinics needed 
to be “led” by NPs, as will be discussed further in Section 4.3 of this report. To maintain and 
accurately portray the current model of care, a few key informants said that the clinics should not 
be called “physician-led.” 

This section of the report describes challenges the clinic has faced in its implementation, 
including administrative issues and the issue of patient complexity. 

4.1.2 Administrative issues 

Electronic Management Records (EMR) system. The EMR system was identified as an 
implementation challenge by a number of key informants. Since some clinic staff members did 
not have experience using an EMR sy'stcm, there was a learning curve involved. The greater 
limitation, however, was with the operation of the system itself. It was not functioning smoothly 
for the first year and a half of the clinic’s operation. The vendor providing llie service was 
selected from a list of possible vendors provided by the Ministry. Since the EMR vendor was not 
located in Sudbury, it was not possible for the clinic and the vendor to interact face to face. Even 
verbal interactions were difficult to achieve because the clinic staff could often not reach the 
vendor by phone. Most of the time, they were directed through a phone loop of recordings. 
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The clinic has since noticed improvements with this situation, and NPs highlighted the benefits 
of having an EMR. system when multiple professionals are providing care to the same individual. 
For example, there would be no need to physically search for a patient’s charts if a colleague had 
it on their desk because all information is available and accessible electronically . 

Interactions with the Ministry. The clinic received good project management support from of 
the Ministry. Clinic management expressed that the Project Manager for the initiative at the 
Ministiy' was helpful and worked well with the clinic. However, clinic management noted that 
delays in receiving budgetary approvals and funding from the Ministry have impacted clinic 
operations. For example, the official hiring of approved staif members is delayed until the clinic 
receives funding for the positions. Delays could be due to the newness of the model according to 
some key informants. Prior to the clinic, there W'ere no guidelines specific for NP-Led clinics. 
The Ministiy' used existing FHT guidelines adapting them as necessary, which took time. 
Determining the forms to be used to apply for finding was also initially unclear. As well, the 
staffing resources dedicated to the NP-Led Clinics initiative at the Ministry were perceived to be 
insufficient by some key informants. Several key informants recommended that the Ministry 
ensure sufficient staffing to assist with the rollout of the next 25 climes. 

Clerical support. Initially, the number of clerical W'orkers (two) at the clinic was inadequate to 
support the clinic’s operations. NPs were often being removed from their direct patient duties to 
perform administrative tasks such as stocking rooms, cleaning examination rooms, or entering 
information into the EMR system. Key informants noted that this lack of clerical support 
affected the rate of patient intake. F unding for a third clerical worker had initially been approved 
by the Ministry, but the clinic did not receive this funding immediately. ITae clinic hired a third 
clerical support worker using separate fiinds in the interim, which alleviated the issue of 
insufficient clerical support. The Ministry is now funding this position. 

Space. The space available at the clinic is insuflicient according to clinic management and the 
clinic s NPs, and initially, finding an appropriate location for the clinic w'as challenging. The 
Ministry and clinic spent a considerable amount of time looking at various potential sites for 
both health clinics, 'fhe Riverside site did not require renovations; however, the Lively site does. 
A guide for space planning has been developed and is being used for improving the Lively 
location. 

On days when both of the physicians are present, there are not enough examination rooms 
available. On these days, NPs have the oppoituniry to conduct administrative work in offices to 
make available examination rooms for use by the physicians. Clinic management and NPs said 
that the current size of the clinic is not large enough for all of the health care providers who were 
planned for the clinic. The clinic’s part-time pharmacist currently rotates desks depending on 
who is not present. The issue of space also presents a challenge in terms of the training of 
medical students and NPs. Clinic management indicated that training could be more effective if 
examination rooms were larger to accommodate more individuals in the same room. 

The opening of the next clinic in Lively will not fully alleviate the concern of insufficient space, 
because the SDNPC will be expanding its team at the same time. Two NPs will be moving from 
the Riverside site to Lively, and they will have two rooms available to them about half the time. 
The Registered Nurse, dietician and social worker will become part of the team when Lively 
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opens. Clinic management said that in some cases, health professionals may not have access to a 
private space to see patients and that the clinic has no available meeting space. 

Clinic nianagement, NPs and Board members said that there is a need for a larger facility. The 
issue of insufficient space, to some extent limits the number of patients that can be registered at 
the clinic. However, a more considerable factor limiting the number of patients at the Riverside 
clinic is patient complexity. 


4.1.3 Patient complexity 

The extent of patients’ medical complexities was unforeseen and continues to affect the clinic in 
a number of ways. According to the 2006 Census from Statistics Canada, the median age for the 
Greater Sudbury region is 41 years (Statistics Canada, 2006)^ The median age for patients of the 
clinic is 40. Although the age of the clinic’s patients does not appear to differ from the age of the 
local general population, many patients of the clinic have complex health issues. A number of 
focus group participants indicated that prior to registering at the clinic, they had not received 
health care in many years because they did not have a family doctor. Most participants had 
previously been going to emergency rooms or walk-in clinics for their primary health care needs. 

Clinic staff, physicians, management and other stakeholders noted the medical complexity of 
patients as well. Many of the clinic’s patients have conditions such as high blood pressiue, high 
blood sugar, or severe coronary artery disease. NPs and physicians said that patients new to the 
clinic often need to visit the clinic frequently in the initi^ period after registering in order to 
“catch up” on and address their health concerns. It is common for new' patients to be seen by an 
NP and a physician for tlie first several appointments. Table 8 provides frequencies with which 
patients who responded to the survey visit the clinic. 


Table 8: Approximately how often do you typically visit the clinic? (Q. 3) 

Frequency of visits 

% of patients 
(n=603) 

Once or more times a week 

1% 

Once a month 

13% 

Every few months (2-5 months) 

48% 

Every 6 months 

18% 

Once a year 

18% 

Don’t know/no response 

3% 

Total percentage may not total to 100% due to rounding. 


The above data indicates that patients arc going to the clinic on a fairly regular basis. 

Table 9 summarizes patients’ reasons for visiting the clinic as determined in the survey. A total 
of 37% of respondents have visited the clinic to monitor their chronic diseases. However, few 
respondents have visited the clinic for pregnancy or child care (4% and 5% respectively). 
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Table 9; For what type of care have you visited the clinic? (Q. 41 

Type of care 

% patients 
(n=603) 

Annual health exams 

86% 

Diaqnosis/treatment of a minor illness 

41% 

Monitoring of a stable chronic disease 

37% 

Referral to specialist 

33% 

Health education 

22% 

Screening for a chronic disease 

19% 

Referral to other health care provider (allied health 
professionals 

18% 

Diagnosis/treatment of a minor injury 

9%“ 

Individual counselling 

9% 

Monitoring of an infant/child's growth and development 

5% 

Pregnancy care 

4% 

Respondents could provide more than one answer; therefore totals sum to more than 1CX)% 


A total of 37% of all survey respondents said that an NP identified something about their health 
that they were previously unaware of. Of the people who had new issues identified about their 
health, 15% said the issue was related to high blood pressure, 13% said tlie issue was related to 
diabetes, and 12% said the issue was related to high cholesterol. Again, these data provide an 
indication of the complexity of health issues of the clinic's patients. 

The complexity of patients appears to be affecting the total number of patients who can be 
registered at the clinic. According to the patient focus groups, clinic management, NPs, and 
physicians, new patients of the clinic may not have received health care in a number of years. It 
is more likely that these individuals would have health issues that have gone undetected for some 
time compared with individuals who receive health care regularly. There is also likely a greater 
chance that there will be multiple issues to address. It is logical that these factors would lead to a 
need for longer visit times, Also, more complex patients require more physician involvement 
because long-standing health issues would require additional medical attention. 

As mentioned in the clinic profile, each NP was initially expected to build a roster of 800 
patients. Since the Ministry^ funds six NP FTE equivalents, this would mean having a total 
patient registry of 4,800. Limited clerical support initially impacted patient intake rates because 
of reasons discussed in Section 4.2.1; however, the main factors currently restricting patient 
intake are space limitations and the complexity of the clinic’s patients. The clinic does not turn 
away patients based on the complexity of their health issues, nor do they intend to screen for 
low'-risk patients. The clinic will not likely meet the target of having 4,800 patients registered by 
the end of three years of operation as planned. 
































Ministry of Health and Long-Term Care 

Evaluation of the Sudbury District Nurse Practitioner Clinics—October 1, 2009 


21 


4,2 Physician integration 

The main issue with respect to physician integration into the clinic is that the current 
compensation model is perceived to be inadequate. Consequently, physicians are not as involved 
in the clinic as they would like to be. The compensation physicians currently receive is 
apparently not competitive with other potential sources of income for physicians in the 
community. Almost ever>' key informant said that this will present a considerable barrier in 
recruiting and retaining physicians for the current and future NP-I.ed clinics being implemented, 
lire clinic is likely to benefit from increased physician input that would come with increa,sed 
compensation. This section of the report describes the current compensation model and discusses 
barriers to physician integration. 

4.2.1 Compensation and consultation 

NPs can arrange for their patients to see a physician directly, or they can consult with a physician 
regarding the health issues of their patients. When a physician sees a patient directly, they bill 
FFS claims to OFIIP for the direct patient encounter. These appointments are usually scheduled 
at 15-minute intervals, NPs will sometimes attend the patient’s first appointment with the 
physician, 

NPs consult with physicians when a situation is beyond their scope of practice or out of their 
comfort zone. For example, an NP would consult with a physician if a patient’s chronic disease 
has destabilized, if abnormal blood tests were received, or for adjusting treatment. Consultations 
are provided formally and informally. Formal consultations benveen physicians and NPs take 
place in fixed time slots at the start of the physicians’ work day. At this time, thirty minutes are 
available for NPs to ask the physician questions. Sometimes, formal consultation extends beyond 
the allotted 30 minutes. Each NP can spend approximately 20-30 minutes per week in forma! 
consultation with physicians. 

Informal consultations arc more fluid, and can take place on or off-site. On-site, NPs may ask the 
physicians a question as they are passing in the hall, or may request that a physician join the 
patient and NP during an appointment. Some NPs share an office with the physicians, which can 
facilitate dialogue. Physicians can also answer the questions of NPs by telephone or email. 
Because of the fluid nature of the informal consultations, physicians and NPs generally found it 
difficult to quantify the amount of time spent consulting informally. Physicians approximated 
that they consult between four and seven hours per week. 

The physicians are compensated for the consultation they provide with monthly stipends. The 
Ontario Medical Association and the Ministry signed an agreement in 2004 stating that 
physicians will receive $800 per month per NP, and this agreement still applies. Since there arc 
six NPs at the clinic, a total of $4,800 is allocated for physician stipends. This is shared between 
the clinic’s two physicians. If the clinic were to hire a third physician, the $4,800 would need to 
he shared among the three physicians. The amounts of the stipends are fixed; physicians are not 
compensated based on the actual amount of time that they spend consulting. Physicians and 
clinic management noted that a 3% raise was recently applied to all FFS in Ontario, but not to 
the NP stipend. ITie clinic has raised frinds to “top-up” physician compensation. 
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NPs said they would like to have more consultation time available. The new site opening in 
Lively will likely lead to greater requirements for consultation according to clinic NPs and 
physicians- Key informants across all groups of stakeholders said that the current compensation 
model is inadequate and needs to be addressed. Tlie current compensation model promotes the 
dynamic that the less time a physician spends in consultation and seeing patients, the more 
lucrative it is for them. It presents several barriers to achieving successful physician integration. 


4.2.2 Barriers to integration 

Separation from the team. The current compensation model does not promote physician 
involvement in the clinic. Key informants across all groups said that the clinic would benefit 
from increased physician involvement. Physicians do not participate in a number of clinic 
activities because the stipend is insufficient. The physicians believe that the clinic would benefit 
if they could play a stronger administrative role. For example, physicians currently do not attend 
monthly clinical team meetings, nor do they participate in hiring clinic staff. There was physician 
participation in developing medical directives for the clinic; however, no compensation wras 
provided for this. 

If resources w'ere available, physicians could offer training in areas w'here knowledge gaps are 
identified. For example, they could offer in-services on certain illnesses. Yet there is no 
opportunity for learning from physicians other than direct consultation, which is limited. In 
addition, physicians do not always have the opportunity to provide suggestions for the clinic and 
arc apparently not always privy to information at the clinic. Key informants recognized that 
having the physicians' input and making the physicians feel like part of the team is essential to 
success; however, the current insufficient remuneration is a major factor hindering their 
involvement and integration. 

Patient complexity. The clinic’s patients appear to have complex health needs, and physicians 
are seeing the patients with the most complex health needs. For example, clinic patients are 
generally not referred to for well-baby checks, pap tests or regular blood pressure tests, but rather 
for more complicated conditions. This has financial and other implications. Patients with more 
complex health issues require longer visits, which can translate to seeing fewer patients per day 
and fewer billable visits. Physicians require more time to become familiar with patient files 
because they are not their regular patients and both physicians are only at the clinic on a part- 
time basis. Physicians see few “repeat patients.” 

Physicians are also not exposed to the diverse array of patients that they would see in their own 
practice. They are not necessarily using all of their skills, and usually see patients with higher 
needs. This can be emotionally draining for them when there is no mix with seeing patients who 
are healthier. 

Referrals. The process of referrals to specialists was seen as inefficient by some key informants, 
but not by others. Specialists will often not accept referrals from NPs because they are not 
remunerated as well as they would be if the referral came from a physician. Physicians will 
therefore often write the referrals themselves or will sign off on referrals written by NPs. Some 
key informants perceived this as inefficient by having the physicians involved unnecessarily. 
How'ever, other key informants preferred the collaborative process of writing referrals. One key 
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informant said that a referral writing workshop provided by a clinic physician would be helpful; 
however, as mentioned previously, physicians are not compensated sufficiently to conduct such 
an activity. 


4.2.3 Potential changes to current compensation model 

A number of key informants said that a salary model for physicians of the NP-Led Clinics could 
help overcome the barriers to collaboration, as long as the salary is sufficient. Some key 
informants noted that salaried positions would be attractive for physicians, who could have the 
option of working reduced hours and with no overhead costs. If FFS compensation is used and if 
the physician is the only team member not on salary, time management is viewed differently and 
a true team approach is not achievable. Appropriate compensation will be necessary to achieve 
successful integration of physicians in the NP-Led Clinic model. However, the Ministry has a 
funding agreement w'hereby Ontario Medical Association (OMA) approval is required for any 
changes to the current funding model. If the NP-Led Clinics initiative is to succeed, then the 
physician compensation model would need to be adapted and negotiated between the Ministiy' 
and the OMA. 


4.3 NP-Led Clinic model 

The clinic model appears to work well overall, with the exception of the issue of insufficient 
physician compensation covered in the previous section of this report. ITiis section describes the 
main benefits and challenges of the NP-Led Clinic model. 


4,3.1 Registration and NP compensation 

The system of patient registration to the clinic as opposed to enrolment to an individual health 
care pro\ ider is seen as a clear benefit of the NP-Led Clinic model because it ensures continued 
access to care for individuals registered with the clinic. Patient registration with the clinic 
represents a key difference between the FHT model and the NP-Led Clinic model. With patient 
enrolment, as in the FHT model, patients enrol with a specific physician by completing an 
enrolment form that is sent to the Ministry. With the NP-Led clinics, patients are not enroled 
with a specific person; rather they are registered with the clinic itseif. If an NP departs from the 
clinic, whether permanently or for vacation, their patients would still have a health care provider 
avail^le to them. Focus group participants expressed that they have a sense of security in 
knowing that they will have continued access to health care. 

The current model of compensating NPs appears to be appropriate. NPs are paid on a salary 
basis, and not on an FFS basis. NPs, Board members, and other stakeholders agreed that this 
mode! of compensation has had a positive impact on the care that is provided. Individual NPs 
have no overhead costs and the amount of time they can spend \vith each patient is not 
determined by FFS. Several focus group participants were awnre that the method of NP 
compensation is salary-based. These participants preferred this type of model instead of a FFS 
model because it removes any correlation between the amounts of time spent with each patient 
and the compensation received. Focus group participants believed that this model of care leads to 
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increased time spent with each patient and inferred that individuals therefore receive additional 
attention and more thorough care. 

The clinic's NPs have professional development opportunities and benefits. At the time of this 
evaluation, key informants indicated that tlie NP-Led Clinics should have access to the Quality 
Improvement and Innovation Partnership (QIIP), a partnership established by the Ministry to 
support the development and implementation of FHTs in Ontario. Recent communication with 
the Ministry has indicated that this is currently underway. 


4.3.2 NP scope and roles 

The clinic model facilitates NPs in functioning to their full scope of practice, which the clinic's 
NPs arc satisfied with. Key informant interviews indicated that prior to working at the clinic, 

NPs were often not practicing to their full scope. Some were not employed; others were 
practicing as Registered Nurses. Some were practicing as NPs, but not to their full scope. The 
e.xperience levels of NPs at the clinic vary. Mentoring opportunities between NPs help new NPs 
to become familiar with their scope sooner than if they were working independently. 

The clinic has developed a set of medical directives to allow the clinic’s NPs to order tests and 
prescribe medications that are beyond their scope. The clinic is also lobbying for an expansion of 
the legislative scope of NPs; NPs identified provincial legislative barriers as impeding them from 
functioning to their full scope. 

Despite limited physician involvement in the clinic, the collaborative relationships between the 
NPs and physicians and among NPs are evident. NPs, physicians, Board members, and clinic 
management indicated that this sense of teamwork at the clinic helps facilitate NPs in 
functioning to their full scope of practice, and the working team environment w'as cited by a 
number of key informants as one of the reasons for the success of the clinic. Patients who 
participated in the focus groups also had positive comments about the ability of clinic NPs and 
physicians to work as a team. 


The roles and responsibilities of NPs are clear to the NPs at the clinic. These responsibilities are 
defined and are accessible to clinic staff on the shared computer drive, along with clinic policies 
and human resources policies. NPs also received hard copies of these documents, and are given 
the opportunity to become familiar with them. However, there are some grey areas. For example, 
the amount of time that physicians should continue seeing patients was not always clear. When is 
the patient ready to return to seeing their NP instead of the physician? In addition, the NP role 
may not always clear to physicians at the clinic. Key informants recommended that training for 
physicians about the role of an NP be provided. However, this w'ould be additional work for 
physicians that again, would not be adequately covered by the current stipend amounts. 


Initially, there was a learning curve in terms of how the roles of different team members are 
played in the run of a day. For example, the clinic’s physicians have different styles of operation, 
which NPs need to learn over time. Similarly, the clinic’s physicians learned that the experience 
levels of NPs at the clinic are varied and communication needs to be varied accordingly. Over 
time, physicians became more familiar with the different levels of experiences of the NPs. Clinic 
team members had to determine in what circumstances the patient should be seen by both an NP 
and a physician. The exact logistics of formal and i nf ormal consultations needed to be defined 
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Key infonnants recommended that such logistics of how a day will flow be discussed at the onset 
of future clinics. 


4.3.3 Accountability and leadership 

Most key infonnants were satisfied with the system of accountability within the clinic. NPs and 
Board members were generally supportive of having the Clinic Director be an NP, because 
knowledge of clinical issues was seen as an essential asset for someone who makes management 
decisions. The logic is that as someone who is familiar with the NP scope of practice, an NP can 
better understand the challenges faced by the clinic than someone who is not an NP. Other key 
informants said although having the Clinic Director be an NP works w^ell for the SDNPC, this 
arrangement may not be necessary in all future NP clinics. Ho’wevcr, these key informants said 
that it would be necessary to have a senior NP or physician responsible for the larger clinical 
decisions. Some key informants noted that having the Clinic Director spend half of the time as an 
NP results in a heavy workload for this individual. As discussed in Section 4.3 of this report, 
increased physician involvement in the clinic was recommended. Physicians could play a 
stronger administrative role in the clinic that could come with a more appropriate compensation 
model. 


The BOD was seen as a strong advantage of the clinic. There is a requirement that 50% of Board 
members be NPs, and a number of key informants expressed the necessity of having NPs serve 
on the clinic s BOD because this increases the Board’s contextual understanding of any issues 
that may arise. However, key informants also highlighted that having a BOD composed of 
people &om diverse backgrounds is advantageous because of the unique, multiple backgrounds 
and mlents brought to the table. Key informants also said that having Board members and the 
Clinic Director be regulated health professionals was a benefit, because it helps ensure adherence 
to certain standards. 

Although the topic of liability was not a prominent issue studied, a couple of key informants 
raised the subject. Because of the model of shared care, with different health care professionals 
providing care to the same individual in different circumstances, it was not clear to all key- 
informants with whom patient accountability rests. Some key informants mentioned that they 
take detailed notes of their patient encounters for this reason. 


4.3.4 Reporting 

The clinic completes reporting forms for the Ministry; however, the usefulness of the current 
reporting system is questionable. 

The clinic submits reports to the Ministiy' that document patient encounters and amounts of time 
spent in consultation with physicians. NPs at the clinic believed that this system of reporting 
does not capture appropriate results. For reporting patient encounters, NPs said that the current 
forms do not allow for the capturing of the complexities of the patients being seen, and described 
the forms as being “too generalized.” NPs also said that it is difficult to report with the physician 
consultation forms because of the fluid nature of the infor ma l consultations. 
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Another major concern about reporting is that the key informants, in particular NPs, are not 
aware of how this information is being used and would Hke to see what is done with the results 
of completed forms. In addition, NPs said that completing the forms is time-consuming and they 
OTc not always instructed on how to complete them. Apparently within the clinic, different 
individuals complete the forms in different ways. Finally one key informant suggested that forms 
could be tailored for individual sites/regions, given that the prevalence of certain health issues 
can vary by region. 


4.4 Patient satisfaction 

Patients are generally highly satisfied with all aspects of the clinic. The high response rate in the 
patient feedback survey was telling. Of the 970 questionnaires mailed out, 603 were received. 
From our experience, a response rate of 62% is excellent and considerably higher than what is 
normally expected for a mail-out survey. With a mail-out survey such as this, we would consider 
a response rate of approximately 30% to be respectable. Along with the high response rate, 
patients showed a high level of satisfaction with the clinic’s services, physical set-up, and 
accessibility. This section describes patient satisfaction in these three areas. 

4.4.1 Services 

Patients showed an overwhelmingly high level of satisfaction with the services they receive from 
their NP, in both the survey and the focus groups. The 2009 Client Satisfaction Survey prepared 
by the clinic BOD also showed high patient satisfaction (SDNPC, 2008a'’). 

In the survey conducted by PRA, a total of 96% of respondents said they were either veiy- 
satisfied or somewhat satisfied with services they received from their NP, Only 2% were 
somewhat or very dissatisfied, and 2% were neutral on the issue. Table 10 summarizes these 
results. 


Table 10: How satisfied are you with the services you receive I 

from your Nurse Practitioner? (Q. 7) 1 

Level of satisfaction 

% patients 
(n=603) 

Very satisfied 

87% 

Somewhat satisfied 

9% 

Neutral 

2% 

Somewhat dissatisfied 

1% 

Very dissatisfied 

1% 

No response 

1% ‘ 

Total percentage may not total to 100% duo to rounding. [ 


The survey results showed satisfaction quantitatively; however, focus groups allowed for a 
greater understanding of the extent to which patients are satisfied and the nature of thei r 
satisfaction. All focus group participants across the three sessions were unanimously satisfied 
with the services provided by their NP. The main reasons for high satisfaction were the attitude 
of the NPs, and the thoroughness of care provided, including health education. 
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Focus group participants were impressed with the “bedside manner” of their NPs. The>' said that 
the excellent listening skills of the clinic’s NPs, along with their caring, friendly and respectful 
approach to health care provision allowed participants to feel comfortable about expressing their 
health issues and asking any questions they have. In addition, participants mentioned that the 
atmosphere in the clinic is a positive one. This approach to health care is one that most 
participants had not experienced with previous health care providers. Participants indicated that 
they were satisfied with the services provided by the clinic’s physicians, and appreciated that 
they have access to a physician if needed. 

The thorough health care that patients receive is another reason for their high satisfaction. Across 
all groups, focus group participants said that NPs are spending more time with them individually 
than did their previous health care providers. They also said they had never before undergone 
such comprehensive health examinations and tests, which they appreciated. Participants also 
valued the consistent follow-up after appointments and tests, and said that if an NP does not 
know an answer to a question, they will find it. As mentioned in Section 4.4.1, several focus 
group participants attributed the thorough care they receive to the fact that NPs are compensated 
based on salary instead of an FFS model. Focus group participants said that the positive attitudes 
and thoroughness of serv'ice provision contribute to addressing the root causes of their health 
issues. Several focus group participants and survey respondents attributed the fact that they are 
alive to the clinic, saying that their health issues could not have gone unchecked and untreated 
for any longer. 

Both survey respondents and focus group participants are pleased with the health education they 
receive from their NP. The sur\'ey indicated that 82% of respondents agree that because of the 
clinic, they feel better prepared to manage their owm health and wellness, and prevent illness and 
worsening of their health. Focus group participants said they were encouraged to ask their NP 
questions if they did not understand something, and that NPs provide them with health education 
and the tools to manage their own health. A couple of participants expressed appreciation for 
being able to view their own medical records. 

A total of 6% (n=36) of survey respondents participated in a program offered by the clinic (either 
Smoking Cessation, HPV Immunization, or Why Weight?}. A total of 72% of these respondents 
were satisfied, 14% were neutral, and 6% were di.ssatisfied with the program they participated in. 
These results are summarized in Table 11. 


Table 11: Overall satisfaction of respondents’ 
participation in clinic programs 

Level of satisfaction 

% 

(n=36) 

Satisfied 

72% 

Neutral 

14% 

Dissatisfied 

6% 

No response 

8% 
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4.4.2 Physical set-up of clinic 

Patients were satisfied with the physical set-up of the clinic, including parking and accessibility 
by public transit. There was also a high level of satisfaction with the comfort of the waiting room 
and examination rooms. 

Table 12 shows the survey respondents’ levels of satisfaction with different aspects of the 
physical set-up of the clinic. 


Table 12: In terms of the physical set-up of the clinic. (Q. 101: 

(n=603} 

% who 
agree 

% who are 
neutraJ 

% who 
disagree 

Don’t know/ 
no response 

1 can always find a seat in the waitinq room 

97% 

1% 

2% 

1% 

The examination rooms are comfortable 

95% 

4% 

1% 

1% 

Paikinq is easy to find 

95% 

1% 

1% 

3% 

The waiting room is comfortable 

90% 

9% 

2% 

<1% 

The clinic is easily accessible by public transit 

65% 

4% 

1% 

30% 

TotaE row percentages may not total to 100% due to rounding. 


Note that in the above table, 30% of survey respondents could not rate accessibility by public 
transit. If "don’t know / no response" is excluded from the results, then 93% of respondents agree 
that the clinic can be easily accessed by public transit. 

4.4.3 Access 

Since the clinic opened in Sudbury, the clinic’s patients believe that their access to health care 
has improved. l^Tien compared with their previous health care situations, participants’ wait times 
have decreased. A condition of registration for potential clinic patients is that they do have a 
health care provider. Therefore, the difference patients perceive in wait times is likely amplified 
by the fact that they previously had no primary health care provider, and were likely attending 
emergency rooms and walk-in clinics for their health care needs. Several key informants pointed 
out anecdotally that the local emergency room was experiencing decreased numbers since the 
opening of the clinic. In the survey, 85% of respondents said that the waiting time for them to see 
an NP is shorter than it used to be to see a doctor. 

Focus group participants said that it is easier to get referred to another health care provider since 
coming to the clinic. This aligns with the survey results; a total of 71% of respondents said it is 
ea.sier to get referred to another health care provider when needed. The patients participating in 
focus groups also supported the idea of having “one-stop-shopping,” with several health care 
services available under one roof. They were interested in seeing the clinic offer even more 
services than it currently has. 

Some focus group participants said that prior to the establishment of the clinic, they were 
traveling outside the Sudbury region for their primaiy' health care. They appreciated that they 
now have access to health care in their own community of Sudbury. Other participants 
appreciated that they no longer needed to attend walk-in clinics or emergency rooms, where 
waiting times can be long and crowded waiting rooms can increase their risk of becoming ill. 
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Several focus group participants suggested that the clinic’s hours of operation could be extended 
into the evening to allow for appointments outside of working hours. This was also a finding in 
the interna! client satisfaction survey conducted by the clinic. However, other participants did not 
believe this was necessary saying people can expect their appointments to begin on time, and can 
typically attend appointments on their lunch hour without having to miss work. Focus group 
participants also commented that it can sometimes be difficult to reach the clinic by phone 
because there is only one line. 

Respondents of the patient feedback suiwey believed that the clinic is an effective way to 
improve the delivery of health care services in Sudburj'. Table 13 summarizes this finding. 


Table 13: Overall, do you think that Nurse Practitioners are an effective way of 
improving the delivery of health care services in Sudbury? (Q. 14) 

Response 

% patients 
(n»603) 

Yes 

95% 

No 

1% 

Don't know/not sure 

3% 

No response 

2% 

TotaJ percentage mav not total to 100% due to roundinq. 


Approximately 77% of survey respondents explained why they do or do not support the above 
statement in the final open-ended question of the survey. These responses were grouped into 
themes and coded, and the main emerging themes are summarized in Table 14. 


Table 14: Overall, do you think that Nurse Practitioners are an effective way of improving health 
care delivery services in Sudbury? Please explain why or why not. (Q. 15) 

Reason provided 

respondents 
(n= 4621 

% 

respondents 

Primary health care otherwise unavailable 

203 

44% 

NPs provide personalized service 

116 

25% 

Visits not rushed/receive detailed explanations 

106 

23% 

More accessible/easy to get to an appointment 

103 

22% 

NPs are knowledgeable, efficient, thorough, effective 

102 

22% 

Physician and specialist referrals are available when necessary 

69 

15% 

Negative comments 

10 

2% 

Total sums to more than 100% because respondents could pro\^de more than one response 


The above table shows that just under half of those who responded to the open-ended question 
said that they support the NP model because primary health care was otherwise unavailable to 
them. 

Of the 462 people who provided an open-ended response, 10 comments were negative. The 10 
negative comments included criticisms about having to pay for missed appointments, the 
strictness of being on time for appointments, a sense of discomfort with the knowledge of the 
NP, and suggestions for increased hours of operation (e.g., evenings). 

Focus group participants said that creating more NP-Led clinics is a good idea, both across the 
province and within Sudbury. The focus group participants were wary about increasing the 
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patient capacity of the SDNPC by too much, fearing that it could compromise the current high 
quality of services they are receiving. 
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5.0 Conclusions and lessons learned 

This section of the report contains the conclusions and lessons learned. A response is presented 
for each evaluation question (sec Table 4, page 12 of this report). 

5.1 Rationale 

The following evaluation question explores the rationale of the clinic. 


1. What considerations led the MoHLTC to develop the Sudbury District NP Clinic? What issue(s; 
vtfas the clinic intended to address? _ 

The clinic was established in Sudbury due to the shortage of physicians in the area and the 
resulting high number of patients with no primary' health care provider. There were NPs living in 
Sudbury who were not employed and who lobbied for the clinic, which led to its establishment. 

I he clinic provides a setting where NPs can function to their full scope of practice, providing 
primary'' health care for patients who did not previously have a health care provider. 

5.2 Design and delivery 

The following evaluation question explores the rationale of the clinic. 


2. Has the clinic faced challenges in its implementation? If yes, what were the challenges and 

how were they overcome? _ 

The clinic experienced implementation challenges with patient complexity, clerical support, the 
EMR system, space and delays in receiving funding, 

► The EMR system is essential and beneficial to the operation of the SDNPC, especially 
given the collaborative approach to health care. Having an EMR system is recommended 
for future NP clinics. Future NP-Led Clinics should ensure that a high-quality EMR 
service provider that is available for communication when issues arise is selected. 

► It would be beneficial for all future NP-Led Clinics to have in place appropriate clerical 
support from the onset so that NPs do not need to take on clerical duties. This has 
impacted operations initially in the case of the SDNPC. 

► Space at the clinic was found to be a limitation, which limits, to some extent, the number 
of patients that can be registered at the clinic. The new clinic opening in Lively will not 
resolve the issue because more staff will be joining the team. Future clinics should ensure 
appropriate space from the onset of operations, ensuring that the facilities they select will 
meet their immediate and future needs. 

► It appears that delays in receiving budget approvals and funding from the Ministry have 
affected the clmic. The Ministry should ensure sufficient staffing and that such delays are 
minimized. 

► The next 25 clinics should learn from the experiences of the SDNPC by communicating 
with the SDNPC about the implementation challenges it faced. The Ministry could 
provide .support for the SDNPC to share information about best practices and lessons 
learned with other clinics. 
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3. Are the current program delivery mechanisms and structure appropriate and effective? _ 

There is a high level of support for the model of delivery of care at the SDNPC. It is focused on 
patients who appreciate the short wait times, thorough care and services they receive. The 
management structure with a Clinical Director as an NP appears to work well and is appropriate 
for the SDNPC. This management structure may not be necessary' in for all future NP-Led 
Clinics. However, for all future clinics, there should be a regulated health professional in charge 
of all clinical operations to ensure that someone with appropriate expertise is involved in 
decisions made. 


4. Does the current reporting system allow for the clinic to adequately communicate ttieir 
_ resuits? 

Tlie current reporting system by the clinic to the Ministry was found to be inappropriate. NPs 
said that the patient encounter forms are too generalized and do not capture patient complexity. 
NPs said the physician consultation forms may not be capturing accurate information because it 
is difficult to document time spent consulting informally since it is so fluid. In addition, no key 
informants were aw'are of how the results are being used. Both the patient reporting forms and 
the consultation reporting forms need revisions so that the information collected is useful. The 
forms should be redesigned with a goal in mind of how the completed forms will be used, and 
individuals completing the forms should receive training on how to do so to ensure consistency 
in data collection. Forms could be tailored for difierent sites depending health issues specific for 
the regions. 


5. What, if any, issues are there with NP/doctor integration? Is the level of physician consultation 

appropriate? _ 

► Physicians are not fully participating as team members in the clinic because the 
compensation model is insufficient and because of the complexity of the clime’s patients. 
It appears that the clinic would benefit from more physician involvement and input, and 
physicians would like to play more of an admini,strative role in the clinic. It is 
recommended that physicians become more involved in the clinic, which would be 
possible with a new compensation model (see Question 9 in this conclusion). 

► There was an initial learning curve where staff at the clinic was learning how roles play 
out in the run of a day. For future clinics, logistics of the way in which clinic team 
members play their roles needs to be discussed at the onset of clinic operations. 

► More physician consultation time would be appropriate. NPs would like to have more 
consultation time available and tire new clinic opening in Lively will likely increase the 
consultation time needed. 


6. Are the NPs functioning to the full scope of their practice? What factors facilitate or impede 
the NPs from functioning to their full scope? __ 

NPs at the clinic are functioning to their foil scope of practice. The nature of the clinic model 
facilitates this because NPs are primary health care providers and therefore deal with a broad 
range of health issues. The clinic has also developed medical directives that facilitate NPs in 
functioning to their foil scope. The collaborative relationship among NPs and between the NPs 
and physicians is another factor facilitating NPs in functioning to their full scope. Future NP-Led 
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Clinics could ensure that individuals hired to work at the clinics have the ability to work in a 
team setting. 


7. Are there any concerns about NP role clarity and scope of practice? If so, how could these 
concerns be mitigated? _ 

There are two main concerns about NP role clarity and scope of practice; 

► The role and scope of practice of NPs at the clinic are clear among NPs. Patients appear 
to understand the basic differences betvreen NPs and physicians. However, the role and 
scope of practice of NPs at the clinic is not always clear to the clinic’s physicians. 
Physicians could be provided with training about the role of NPs at the clinic. 

► Although patients of the clinic generally understood the role of their NP, the general 
public may not. There is an apparent need to increase public understanding about the role 
of an NP and the NP-Led Clinics initiative in general. The negative media attention 
generated by physicians questioned the effectiveness, quality of care received, and 
potentid costs of the clinics. This may have created a lack of confidence in the clinic and 
the initiative, and a sense that there is dissention among health care workers. Negative 
media attention may occur in other areas where NP-Lcd Climes are being implemented. 
A provincial media campaign led by the Ministry' was recommended to increase public 
understanding about the initiative. The campaign should focus on educating the public 
about how health services are delivered in this type of model, including the roles of NPs 
and physicians at the clinics. 


8. Is the MP woricload appropriate? What factors affect their workload? _ 

The current w'orkload of NPs at the clinic appears to be appropriate, which indicates that the 
current number of patients registered with the clinic is appropriate. The opening of the new site 
in Lively is expected to increase the number of patients registered with the clinic to some extent; 
however, it will not likely be possible to reach the goal of 4,800 patients by August 2010 while ' 
maintaining the quality of sendees h is currently providing. This is in part due to the fact that 
many of the clinic’s patients have complex health issues, requiring more time and attention from 
both NPs and physicians. Increasing registration to meet the target number would likely have an 
impact on the qualities of the clinic that patients believe are so beneficial, such as thorough care 
and short wait times. The Ministry can look to the capacity of the SDNPC when planning for 
future clinics to estimate the number of patients that can be served. 

9. Is the current physician compensation model appropriate? Why or why not? 

The current physician compensation model is not appropriate, and was identified as the most 
serious concern in this evaluation. If the model is not changed, there wdll be considerable 
challenges in implementing the next 25 NP-Led Clinics because it will be difficult to recruit and 
retain phy sicians. The current compensation model is apparently not competitive with other 
potential sources of income for physicians and it limits physician involvement and input into the 
clinic, which hinders collaboration. Providing physician remuneration in the form of a salary- 
based model or hourly wages i.s a possible solution. If the next 25 NP-Led Clinics are to succeed, 
a new model of compensation needs to be established and negotiations between the Ministry and 
the OMA would be necessary. 
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5.3 Satisfaction 

The following two questions address satisfaction. 


10. How satisfied are patients with the services provided by the NPs? How satisfied are the 
patients with the cMnic itself? _ 

Paticnte are generally highly satisfied with the ser\'iccs they receive from the clinic and with the 
clinic itself. The high survey response rate in itself indicates a high level of interest in providing 
feedback about the clinic. Given the overall positive responses in the survey, it can be inferred 
from the high response rate that patients were keen to express their support for the clinic. They 
appreciate the positive and respectful attitude of their NP, and are comfortable asking questions. 
Patients also appreciated having access to physicians when needed, and are satisfied with the 
thorough care they receive. They appreciate the amount of time that is spent with them per visit 
and the good listening skills of their NP. 


11. How satisfied are the HPs, doctors, and other stakeholders with the clinic? _ 

The NPs, doctors, and other stakeholders who were interviewed generally show’cd a high level of 
satisfaction with the clinic. NPs appreciate being able to function to their full scope efficiently 
and in an empowering setting. The clinic’s doctors are satisfied with the model and setting with 
the exception of the current physician compensation model, which is not appropriate. Both 
physicians would like to play a stronger administrative role. Because of insufficient 
remuneration, physicians are not as involved in the clinic as they would like to be. Key 
informants from all stakeholder groups agreed that more physician involvement would benefit 
the clinic. NPs, doctors, and other stakeholders are generally satisfied with the leadership at the 
clinic, expressing that strong leadership with dedicated team members was a necessary 
component for the clinic’s success. Such leadership wall be a key asset in ensuring the success of 
future clinics. 


5.4 Impact 

1 he final two evaluation questions respond to the issue of impact. 

12. Do patients perceive any differences in primary health care in their communities since the 
implementation of the Sudbury District NP Ctinic? _ 

Patients registered at the clinic perceive major differences in the primary health care they receive 
since the clinic was opened, especially given that prior to the clinic, they had no primary health 
care provider. They were largely attending walk-in clinics and emergency rooms for their health 
needs. Because of the clinic, they experience shorter wait times and feel better prepared to 
manage their own health. It is likely that NP-Led Clinics in other areas of the province could 
have a similar impact. 


13. What impact has the clinic had on access to primary health care? _ 

By providing primary health care access to individuals who previously had none, the clinic 
appears to be an effective way to improve access to primary health care in Sudbury. 
















Ministry of Health and Long-Term Care 

Evaluation of the Sudbury District Nurse Practitioner Clinics—October 1, 2069 


35 


References 

1 . College of Nurses of Ontario. (2009). Practice Standard; Nurse Practitioners. Pub. No. 
41038, 'Foronto, Ontario. Retrieved on September 29, 2009, from 
http ://wvw.cno. org/docs/prac/41038_StrdRnec.pdf 


2. Government of Ontario. (2009, June). Introduction to Nurse Practitioner-Led Climes. 
Nurse Practitioner-Led Clinics: Advancing Family Health Care. 
http://www.heaIth.gov.on.ca/transformatiorL/np_clinics/guides/np_clirac intro.pdf 

3. Government of Ontario. (2008). Status of Nurse-Led Clinics - Update - November 21 
2008. 

4. Myers, L. (2009, January 9). Lively doctors concerned about nurse practitioner clinic. 
Northern Life. Retrieved on October 1, 2009, from 
http://www.northemlife.ca/news/localNew's/2009/doctors090109012.aspx 

5. Romanow, R. J. (2002), Building on values: the future of health care in Canada. 
Commission on the future of health care in Canada. Saskatoon, SK. 


6. SDNPC. (2008a). S0NPC Client satisfaction sur\^ey. Prepared by SDNPC Board of 
Directors. Sudbury, Ontario. 

7. SDNPC. (2008b). Our patients’ comments. Retrieved on August 17, 2009, from 
http://sdnpc.ca/client-testimonia1s/ 

8. The Sudbury Star. (2009a). Another silo of health care. Letters to the Editor, Sudbury, 
Ontario. Published on January 20, 2009. 

9. Statistics Canada. (2006). 2006 Sudbury and District Health Unit Community profiles. 
Retrieved on August 12, 2009 from, 

http ://www 12. statcan .ca/englisfr'ccnsus06/data/profi les/community/ 



APPENDIX A 

Interview Guides 



Ministry of Health and Long-Term Care 

Evaluation of the Sudbury District Nurse Practitioner Clinics—October 1,2009 


1 


Evaluation of the Sudbury District Nurse Practitioner Clinics 
interview Guide for Doctoi^ 

The government of Ontario is establishing 25 Nurse Practitioner-Ied clinics, aiming to improve 
access to fanrily health care services in the province. The Sudbury District NP-led Clinic, the 
first in the province, was opened in August 2007 as a three-year pilot project The Ontario 
Ministry of Health and Long-Term Care has engaged PRA Inc. to conduct an evaluation of the 
clinic. The evaluation will support the ongoing operation of the clinic in Sudbury, and will 
inform the implementation process for the 25 new clinics being established in other regions of 
Ontario, 

As part of the evaluation process, we are conducting mter\dews with a number of key 
informants. The information you provide is for research purposes only and will be administered 
in accordance with the Privacy Act and any other applicable privacy laws. The information 
collected in this interview is confidential, and no individuals wdll be identified in any report 
resulting from this study. 

Background 

1. Please describe your experiences in the field of health serxices. When did you become 
involved with the Sudbury District Nurse Practitioner Clinic? 

Rationale 

2. What are the main issues that the Sudbury District Nurse Practitioner Clinic is intended to 
address? 

Design and delivery 

3. Do you believe that your role as a doctor at the clinic is clear? Do you believe the roles 
and responsibilities of the NPs, clinic management, and other clinic staff are clearly 
defined? 

4. Are the Nurse Practitioners at the clinic functioning to their full scope of Registered 
Nurse in the Extended Class practice? If j'es, what factors have facilitated their 
functioning to the full scope of their practice? If not, what factors have impeded their 
functioning to the full scope of their practice? 

5. What are the main reasons that Nurse Practitioners consult with you? How much time do 
you spend in formal consultation with Nurse Practitioners (per day/week/month)? In your 
view, is this an appropriate amount of time? 

6. Please describe the process bj'^ which Nurse Practitioners refer patients to you. Do the 
Nurse Practitioners refer patients to other health professionals? To specialists? If yes, 
please describe these processes. 

7. Is the current shared care model appropriate? What, if any, changes would you make to 
the model? 
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8. Is the current model for compensating doctors appropriate? Are any changes needed to 
this model? 

Satisfaction 

9. How satisfied are you with the clinic? What have you observed about patient satisfaction 
with the clinic? 

Impact 

10. What lessons have been learned from your experiences? In your opinion, what w^cre the 
elements of success in implementing the clinic? 

11. ^at advice would you give to the Ministry of Health and Long-Term Care in 
implementing the next 25 clinics in Ontario? 

Conclusion 

12. Is there anything else you would like to add? 


Thank you for your participation. 
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Evaluation of the Sudbury District Nurse Practitioner Clinics 
Interview Guide for Nurse Practitioners 

ITie government of Ontario is establishing 25 Nurse Practitioner-led clinics, aiming to improve 
access to family health care services in the province. As you are aware, the Sudburj^ District NP- 
led Clinic, the first in the province, was opened in August 2007 as a three-year pilot project. The 
Ontario Ministry of Health and Long-Term Care has engaged PRA Inc. to conduct an evaluation 
of the clinic. The evaluation will support the ongoing operation of the clinic in Sudbury, and will 
inform the implementation process for the new 25 clinics being established in other regions of 
Ontario. 

As part of the evaluation process, we are conducting interviews with a number of key 
informants. The information you provide is for research purposes only and will be administered 
in accordance wqth the Privacy Act and any other applicable privacy laws. The information 
collected in this intervdew is confidential, and no individuals will be identified in any report 
resulting from this study. 

Background 

1. Please describe your experiences in the field of health services. When did you become 
involved with the Sudbury' District Nurse Practitioner Clinic? 

Rationale 

2. What are the main issues that the Sudbuiy' District Nurse Practitioner Clinic is intended to 
address? 

Design and delivery 

3. Please describe the management structure of the clinic. Do you think the structure is 
appropriate? 

4. Are you aware of any challenges that the clinic has faced in its implementation? If yes, 
what were these challenges, and how were they resolved? 

5. Do you believe that your role as an NP at the clinic is clear? Do you believe the roles and 
responsibilities of clinic management, doctors, and other clinic staff are clearly defined? 

6. Arc you functioning to the full scope of Registered Nurse in the Extended Class practice? 
If yes, what factors have facilitated your functioning to the full scope of practice? If not, 
what factors have impeded your functioning to the frill scope of practice? [Do you believe 
there is a clear understanding of the role of an NP among patients? Among the 
community of Sudbury?] 

7. What is the size of your w'orkload? What is your client caseload? Would you be able to 
take on any more patients than you currently have? [Are you satisfied with the 

registration process?] 
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8 . 


9. 


10 . 


What are the main reasons for consulting with doctors? How much time do doctors spend 
in formal consultation with you (per day/week/month)? In your view, is this an 
appropriate amount of time? 

Please describe the process of referring patients to doctors. Do you refer patients to other 
health professionals? 1 o specialists? If yes, please describe these processes. 

Is the current share care model appropriate? WTiat, if any, changes wnuld you make to the 
model? 


11 . 


Is the current model for compensating doctors appropriate? Are any changes needed to 
this model? 


12. Does the current reporting system capture appropriate information? Does it allow for 
good communication of results? 

Satisfaction 

13. How satisfied are you with the clinic? What have you observed about patient satisfaction 
with the clinic? 


Impact 


14. 


15. 


16. 


Based on your observations, does the community being served perceive any changes in 
access to primary' health care? 

WTiat lessons have been learned from your experiences? In your opinion, what were the 
elements of success in implementing the clinic? 

What advice would you give to the Ministry' of Health and Long-Term Care in 
implementing the next 25 climes in Ontario? 


Conclusion 

17. Is there anything else you would like to add? 


Thank you for your participation. 
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Evaluation of the Sudbury District Nurse Practitioner Clinics 
Interview Guide for Board Members 

The government of Ontario is establishing 25 Nurse Practitioner-led (NP) clinics, aiming to 
improve access to family health care ser\'ices in the province. The Sudbury District NP-led 
Clinic (SDNPC) opened in August 2007 as a 3-year pilot project. Tlie Ontario Ministry of Health 
and Long-Term Care has engaged PRA Inc. to conduct an evaluation of the clinic. The 
evaluation w!l support the ongoing operation of the clinic in Sudbury, and will inform the 
implementation process for the new 25 clinics being established in other regions of Ontario. 

As part of the evaluation process, we are conducting interviews with a number of key 
informants. The information you provide is for research purposes only and will be administered 
in accordance with the Privacy Act and any other applicable privacy laws. The information 
collected in this interview is confidential, and no individuals will be identified in any report 
resulting from this study. 

Background 

1. What is your role on the SDNPC Board of Directors? How long have you serv'cd on the 
Board? 

2. Please briefly describe the mandate and activities of the Board of Directors. 

3. How was the Board involved in the implementation of the Sudbury District Nurse 
Practitioner Clinics? 

Rationale 

4. What are the main issues that the Sudbury District Nurse Practitioner Clinic is intended to 
addre.ss? 

5. In your opinion, wfiat are the benefits and/or drawbacks of NP-led clinics, when 
compared with physician-led clinics? 

Application 

6. Was the Board of Directors involved in the application process? Were you involved in 
the application process? 

7. If yes, how satisfied were you with the process of applying to the Ministry for binding for 
the clinic? 

Design and delivery 

8. Has the Board of Directors been beneficial for the clinic? Please explain. [Are there any 
changes you would make in the functioning of the BOD?] 

9. Please describe the management structure of the clinic. Do you think the structure is 
appropriate? 

10. Is the current model for compensating doctors appropriate? Are any changes needed to 
this model? Please explain. 

11. Please describe the current reporting system. Does this system capture appropriate 
information? Does it allow for good communication of results? 
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Implementation 

12. Are you aware of any challenges that the clinic has faced in its implementation? If yes, 
what were these challenges, and how were they resolved? 

13. Is the amount of funding the clinic receives from the Ministry appropriate for the clinic’s 
needs? 

14. What challenges, if any, do you foresee in the implementation of the 25 NP-led clinics in 
Ontario? 

Impact 

15. How satisfied arc you with the clinic? What have you observed about patient satisfaction 
with the clinic? 

16. Based on your observations, doe.s the community being served perceive any changes in 
access to primary' health care? 

17. Are you aware of any other provinces or countries where NP-led clinics have been 
implemented? If yes, please describe. 

18. What lessons have been learned from your experiences? In your opinion, what were the 
elements of success in implementing the clinic? 

19. What advice would you give to the Ministry of Health and Long-Temi Care in 
implementing the next 25 clinics in Ontario? 

Conclusion 

20. Is there anything else you would like to add? 


Thank you for your participation. 
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Evaluation of the Sudbury District Nurse Practitioner Clinics 
Interview Guide for the Ministry of Health and Long-Term Care 

As you are aware, the government of Ontario is establishing 25 Nurse Practitioner-led (NP) 
clinics, aiming to improve access to family health care services in the province. The Sudbury 
District NP-led Clinic opened in August 2007 as a 3-year pilot project. The Ontario Ministry' of 
Health and Long-Term Care has engaged PRA Inc. to conduct an evaluation of the clinic. The 
evaluation will support the ongoing operation of the clinic in Sudbury, and will inform the 
implementation process for the 25 new clinics being established in other regions of Ontario. 

As part of the evaluation process, we are conducting interviews with a number of key 
informants. The information you provide is for research purposes only and will be administered 
in accordance with the Privacy Act and any other applicable privacy laws. The information 
collected in this interview' is confidential, and no individuals will be identified in any report 
resulting from this study. 

Background 

1. Please describe your background in the field of health and your position in Ontario's 
Ministiy of Health and Long-Term Care. 

2. Please describe y'our involvement wnth the NP-Ied Clinic pilot project in Sudbury, and 
your involvement with the larger project to implement 25 NP-lcd clinics In Ontario. 

Rationale 

3. What factors led to the Mimstry's decision to proceed with the establishment of the first 
Sudbury District Nurse Practitioner Clinics and of the next 25 NP-led Clinics in Ontario? 

4. To your knowledge, to what extent has the decision to implement the NP-led Clinics in 
Ontario been supported by other health professionals, and the public? 

Implementation 

5. How were the target areas for the establishment of NP-!ed clinics selected? 

6. Please describe the application and selection processes for determining clinic locations. 

7. What is the current stage of implementation for this project? How' many applications for 
the establishment of NP-led clinics has the Ministry' received at this point? 

8. Please describe how' the clinics will be funded, including levels of immediate and long¬ 
term support provided by the Ministry. 

9. What challenges, if any, do you foresee in the implementation of the 25 NP-led clinics? 

10. Please provide any comments you may have on the lessons learned from the Sudbury 
District NP Clinics pilot project. 

Conclusion 

11. Is there anything else you would like to add? 


Thank you for your participation. 
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Evaluation of the Sudbury District Nurse Practitioner Clinics 
Interview Guide for Stakeholders 

I he government of Ontario is establishing 25 Nurse Practitioner-led (NP) clinics, aiming to 
improve access to family health care services in the province. The Sudbury District NP-led 
Clinic opened in August 2007 as a 3-year pilot project. The Ontario Ministry of Health and 
T^ong-Teim Care has engaged PRA Inc. to conduct an evaluation of the clinic. The evaluation 
wall support the ongoing operation of the clinic in Sudbmy, and will inform the implementation 
process for the new 25 clinics being established in other regions of Ontario. 


As part of the evaluation process, we are conducting interviews with a number of key 
informants. Tire information you provide is for research puiposes only and will be administered 
in accordance with the Privacy Act and any other applicable privacy laws. The information 
collected in this interview is confidential, and no individuals will be identified in any report 
resulting from this study. 

Background 

1. Please briefly describe the mandate and activities of your organization. 

2. Was your organization involved in the implementation of the Sudbury' District Nurse 
Practitioner Clinics? If yes, please describe your invoh^ement. 

3. Has your organization been involved with the larger project to implement 25 NP-led 
clinics in Ontario? If yes, please describe y'our involvement. 

Rationale 

4. What are the main issues that the Sudbury District Nurse Practitioner Clinic is intended to 
address? 

5- In your opinion, what are the benefits and/or drawbacks of NP-led clinics, when 
compared with physician-led clinics? 


Implementation 

6. What challenges, if any, do you foresee in the implementation of the 25 NP-led clinks in 
Ontario? 

7. Are you aware of any other cases or regions where NP-led clinics have been 
implemented? If yes, please describe. 

8. Do you see any alternatives to NP-led clinics that could be used by the Ministiy of Health 
and Long-Term Care to address the lack of health care services in underserviced regions 
of the province? 

9. What advice would you give to the Ministry of Health and Long-Term Care as they 
proceed with the implementation of the next 25 NP-led clinics in Ontario? 


Conclusion 

10. Is there anything else you would like to add? 


Thank you for your participation. 
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Evaiuatton of the Sudbury District Nurse Practitioner Clinics 
Guide for Patient Focus Groups 


Hello everv'one. Our research company, PRA Inc., has been hired to evaluate the Sudbuiy^ 
District Nurse Practitioner Clinics. As part of this study, we are holding focus groups with 
patients of the clinic. We'd like to hear your opinion of the clinic, including its strengths and 
weaknesses, and suggestions for improvement that you might have. 

We are going to discuss these issues over the next hour. We are interested in everyone’s 
opinions; everyone will get a chance to speak. 

Your participation in this focus group is voluntary and confidential. Your comments will not be 
linked to you, but we will write a summary of what even^one in the group said. Nothing you say- 
today will affect the services you get in this community or anj-where else. 

We are tape-recording this discussion only so that we can make accurate notes afterwards. Does 
anyone have any questions before w’e start? 


1. [Ask each person to introduce themselves and say how long they have lived in Sudhury.'] 

2. How' did you find out about the Sudbury District Nurse Practitioner Clinics? 

3. Ts the role of a Nurse Practitioner clear to you? 

4. Has your access to healtlt services changed since the clinic opened? If so, how? 

5. How' satisfied are you with the services you receive from your Nurse Practitioner? 

6. How' satisfied are you with the clinic (for example, the set-up of the building, hours of 
operation, wait-times)? 

7. How' satisfied are you with the referral process the Nurse Practitioners use (i.e., referral to 
doctors and other health providers)? 

8. The Ministry of Health and Long-Term Care is in the process of implementing another 25 
Nurse Practitioner-led clinics in Northern Ontario. Is there any advdee you would give the 
Ministry as they implement these clinics? 

9. Is there anything else you would like to add? 


Thank you for your participation.. 
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Sudbury District 
Nurse Practitioner Clinics 

359 Riverside Drive, Suite 107, Sudbury, ON P3E 1H5 

Phone 705-671-1661 Fax 705-671-0177 Internet www.sdnpc.ca 


May 12,2009 
Dear Sir or Madam, 

We would like to Invite you to take part in this pre text survey about the Sudbury District Nurse 
Practitioner Ginic, The Ontario Ministry of Health and Long-Term Care is conducting a standard 
evaluation ot the clirric. The Ministry is proceeding with Its plan to impfement another 25 Nurse 
Practitioner {WP)-*ed clinics in Ontario, and this evafuatian will help inform that process. 

As a patient of the Sudbury District Nurse Practitioner Clinic, we are interested in hearing your views. 
This confidential survey is intended to obtain feedback about your satisfaction with the cfinic. The 
important information you and others provide will help the clinic to identify areas for improvement, 
and wiit provide useful information for the future NP-3ed clinks in Ontark). The questionnaire should 
only take about 15 minutes to complete, and a pre-pa id return envetope is enclosed for you to return 
the questioofiaire. 

You are one of ten people being asked ta oompEete this questionnaire in advance. Your comments will 
be used to help improve the survey. As you complete the questionnaire, please make a note beside any 
questions you find confusing or unclear, and explain what is uriclear. En addition, the final question 
provides space for you to make additional comments about the survey and off^ any suggestions for its 
improvement 

Your participation Is entirely voluntary. We hope you will participate and provide as much information 
as possible. We want to give you every opportunity to participate in this study. Your answers will be 
kept strictly confidential and will be opmblned with those of others in the final repoix Individual survey 
answers will not be shared with anyone. We would appreciate it If you couid take the time now to 
complete and return yaur questionnaire. 

To protect your privacy, all questionnaires were sent directly from the clinic and your contact 
information has not been shared with any outside party. This survey is anonymous; you do not need to 
indude your name. 

To manage the survey process and also to ensure confidentiality, the Ministry has engaged the services 
of PRA (Prairie Research Associates! Inc. PRA 1$ an indepertdent, national research firm that ts under 
contract to the Ministry. 

If you would like more information about the survey, or have questions on how to complete the 
questionnaire, please do not hesitate to call Sarah Fraser of PRA at 1-S66-422-846S (tolkfreel 

Thank you in advance for your participation! 



Marilyn Butcher 
Gifiic Director 

Sudbury District Nurse Prsrtitioner Clinic 


r'ijM c.' It 09 Pretest tetter cnglisli.rf^ioc 


ysdtr^-^rij^ S)1W 
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Sudbury District 
Nurse Practitioner Clinics 


359 Riverside Drive, Suite 107, Sudbury, ON P3E IMS 

Phone 705-671-1661 1^x705-671-0177 Internet vvww.sdnpc.ca 

Mav 12, 20C9 
Monsieur / 

Wous airrserions vous invfter & partldper au pn^^test du sondage portant sur la c'lnlqu® Sudbury District 
Nurse Practitioner, taquelie fait actueHament f'objet «fune evaluation de la part du mk^istbr^ de la 
et des SoTns de longue dur^ de ^^OnT3lrio. Cette evaluation s'^in^crit dans le cadre du plan du 
Minrst^re qui vise ^ cr^er vingt-cinq nouvelles clintques dirii^ par et irtfirmJw^ 

pradciens en Ontario. L'^valLiatlon servira i orienter le processus. 

Nous sotihaitons oonnaftre votre optnian en tant que patient de la clinique Sudbury District Wurse 
Practitioner. Ce sondage confidentiel a pour bur d'obtenir votre retroaction quart a votre sattsfaction 
par rap pert I la dinique. Tous les renseEgnements que vous et les autf?? participants partagere? 
fourniront des pistes d'am^Uoration ei fourniront de i'Enfornnatlon utile pour ies futures cSnEques 
dingoes par des I'ir^firTnieriereJs praticien(ne)s en Ontario. Environ 15 minutes dtvraient suffjre pour 
rempllr !e questionnaire. Pri^e de nous le retourner dans Tenveloppe pr^affranchie cr jomte. 

Vous Stes 1'une des dix personnes que nous avons invitees I remplir le questionnaire avant sa 
distribution genefale. Vos commentaires permanront d'am^iorer le sondage En remplissant le 
questionnarre. veuillet noter les questions qui poitent ^ confusion ou que uous ne cdmprenej pas, et 
expliquez-en les raisons. De plus, la deml^ que^lCMi uous permet d’^mettre d'autres commentalres et 
suggestions qui nous permettront d'am^ljorer le sondage. 

Votre paTtkrpation est entidrement vobntalre. f^ous esp4rons qtie vous participerez et que vous nous 
fournirez le plus d'information possible. Nous souhaitons fadiitcr votre participation a notre^ude. Vos 
r^onses demeureront confidentielles et seront combin^es a celles des autre* psrtidpants dans noire 
rapport ffnaL Aucune riponse indryiduelle ne sera partegee avec quI que oe solt Nous vous serions 
reconnalssants de bien vouloEr prendre le temps de remnlSr le questionnaire et nous le retourner dsns 
les plus brefs d^ais. 

Aftn de proteger votre anenymat, tous les questionnaires ont envoyfs dlrecrement de la dinique. 
Aucune tierce partie ne connalt vos cPordonn^es. Le sondage est confidentiel : voos n'avei pas a 
foumir votre nom. 

Pour gerer le processus et assurer la confIdentlalEte des renseignernents, !e MfnTstfere a fait appd ^ PRA 
(Prairie Research Associates I Inc. PRA est une entreprise de recherctie nationale et irdi^pendante, 
engagfe par le Ministere. 

Si VO us souhaitez obtenir plus de renseignements sur le sondage, ou si vous avez ties questions quant g 
la fsgon de rmpifr le questionnaire, n'hesrtez pas a communlquer avec Sarah Fraser de PRA au 1 S66 
422*8468 [sans frais). 

Nous vous remercions k favance de votre coWaborat'on! 

Cordialement v6tm. 



Marilyn Butcher 

Directrice de la dinique Sudbtjuy District Nurse Practitioner 


12 0^ Pretest InttjrfripfttKdooi 


VT.V U,23» 



I 
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Evaluation of the Sudbury District Nurse Practitioner Clinic 

Patient Feedback Survey 


Please take a few minutes to complete this survey. When you have finished, please mail it in 
the included pre-stamped, pre-addressed envelope. Your response is confidential. 


tf you are completing this survey based on your 
own experiences ... 

In what year were you born?_ 


If you are completing this survey on behalf of 
someone else (i.e., elderly parent, etc.) please 
indicate ... 


What is your gender? ±i Male ±2 Female i, Patient's age_ 

____ _ . i _ Patient’s gender +t Male ±2 Female 


1. How did you first find out about this health 
clinic? 

±01 Family physician referred me 

±02 Another health professional referred me 

+03 Community newsletter/newspaper 

+04 Received information In the mail 

+0S Friend or family member 

±os Public information display or booth 

±i)7 Community meeting 

±08 Advertisement 

+09 Online 

±66 Other (specffy) _ 


2. How long have you been coming to the clinic? 

±01 1-6 months 

±02 6-12 months 

±03 12-18 months 

±04 More than 18 months 

±88 Don't know/can’t recall 

3. Approximately how often do you typically visit 
the clinic? 

±01 Once a year 

±02 Every 6 months 

±03 Every few (2 to 5) months 

±04 Once a month 

+05 Once a week 

±0$ More than once a week 

±88 Don’t know/can't recall 


4. For what types of care have you visited the 
clinic? {Check ail that apply) 

Diagnosis/treatment of a minor illness 
(e.g., cold, ear infection) 

±02 Diagnosis/treatment of a minor injury 
{e.g., sprain, cut, scrape) 

+03 Screening for a chronic disease 
(e.g., diabetes) 

±04 Monitoring of a stable chronic disease 

(e.g,, diabetes, asthma, high blood pressure) 

±05 Pregnancy care 

±06 Monitoring of infant / child growth and 
development 

±07 Annual health exams 
(e g., complete physical) 

±08 Health education 

(e g,, about your medications, your condition) 

±09 individual counselling 

(e g., diet, smoking cessation) 

±10 Group counselling / teaching activities 

(e.g., pre-natal classes, sexual health clinic) 

±11 Referral to specialist 
(e.g,, cardiologist) 

±12 Referral to other health provider 
(e.g., physiotherapist) 

±66 Other (specily)_ 


iNSTRUCTIONS: Please read each question carefully and check (V) or write in the appropriate response 
When you have finished, return your completed questionnaire by mailing it back in the pre-addressed envelope. 





















CONFIDENTIAL WHEN COMPLETED 


5. Has anyone ever explained the role of the 
Nurse Practitioner you are seeing at the clinic? 
Yes. the Nurse Practitioner explained it 
Yes, a physician explained it 
Yes, someone else explained it 

(specify who) _ 

No one explained it 
Don't know/can't recall 

There are differences between the role of a 
Nurse Practitioner and the role of a family 
physician. Do you believe you understand what 
these differences are? 

Yes 
No 

Don’t know/unsure 

How satisfied are you with the services you 
receive from your Nurse Practitioner? 

Very satisfied (Go to Question 9) 

Somewhat satisfied {Go to Question 9) 

Neutral {neither satisfied nor dissatisfied) 

(Go to Question 9) 

Somewhat dissatisfied (Go to Question 8) 

Very dissatisfied (Go to Question 8) 

8. If you were dissatisfied with the services you 
received, piease explain why. 


9. Since coming to this clinic, did your Nurse 
Practitioner identify anything about your 
health that you were previously unaware of? 
±01 Yes (please identify the condition) 


±00 No 

±ss Don’t know/not sure 


10. Please indicate your level of agreement with 
each of the following statements. 


in terms of the 
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Parking is easy to find... 

_±5 

±4 


±2 

±1 

The clinic is easily 
accessible by public 
transit. 

±5 

±4 

±3 

±2 

±1 

1 can always find a seat 
in the waiting room. 

is 

±4 

±3 

±2 

±1 

The waiting room is 
comfortable. 
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±2 

±1 

The examination 
rooms are comfortable. 

±5 
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±3 

±2 

±1 


11. Please indicate your level of agreement with 
each of the following statements. 
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services has improved for 
residents of Sudbury. 

±5 

±4 

±3 

^2 

±1 

The waib'ng time to see 
the Nurse Practkioner is 
shorter than it used to be 
to see a doctor....... 

±5 

±4 

±3 

±2 

±1 

!t IS easier to get referred 
to another health provider 
(e.g., physiotherapist) 
when i need it.... 

±5 

±4 

±3 

±2 

±1 




f feel better prepared to care 
for mysetf and to manage my 
own health and wellness. 

±5 

±4 


±2 


1 feel better prepared to 
prevent illness / 

±5 

±4 

±3 

±2 

±1 


worsening of my health 


question carefully and check or write in the appropriate response. 
When you have finished, return your completed questionnaire by mailing it back in the pre-addressed envelope. 
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12. Have you ever participated in any of the clinic 
programs? (i.e., Smoking Cessation Program, 
HPV Immunization Program, or Why Weight?) 

±01 Yes 

±00 No (Go to Question 14) 

±ae Don't know/not sure (Go to Question 14) 

13, Please indicate your level of satisfaction with 
the programs that you have participated in. 


14. Overall, do you think that Nurse Practitioners 
are an effective way of improving the delivery 
of health care services in Sudbury? 

±01 Yes 

±00 No 

±Ba Don't know/not sure 
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Cessation 

Program.. 


*0 

4-J 



<u 

iA 







.£2 



to .32 

'iS 



X ^ 

(Q 

(0 


2 

^ <0 

b 

E 


£ S2 


o 

o 

o 

> 


z 



■n 



ts ±4 ±3 +2 ±1 


0) 

g. 

o 

■E 

ec 

a 

’S 

c 

;o 

Q 


to 


15. Please explain why or why not 


HPV 

Immunization ±a ±o ±2 ±1 

Program... 


Why Weight 
Program. 


±5 ±4 ±3 ±2 ±1 ±0 





Thank you very much for your contribution to this research. 

Please send your completed questionnaire by July 17, 2009 by mail 
using the pre-stamped and pre-addressed envelope to; 

PRA Inc. 

500-363 Broadway 
Winnipeg, Manitoba R3C 3N9 





INSTRUCTIONS: Please read each question carefully and check {^) or write in the appropriate response. 
When you have finished, return your completed questionnaire by mailing it back in the pre-addressed envelope. 


































Sondage aupres des patients dans le cadre de revaluation de la 
Clinique Sudbury District Nurse Practitioner 


Veuillez prendre quelques minutes pour remplir ce questionnaire. Veutilez retoumer !e 
questionnaire dumenf rempli dans I’enveloppe preaffranchie ci-jointe. Vos reponses demeureront 

confidentielles. 

-------- / 


Si vous repondez aux questions en fonction de 
VOS propres experiences ,.. 

En quelle annee etes-vous ne(e)?_ 

Vousetes.., ±, un homme ±2 unefemme 


Si vous repondez aux questions pour quelqu'un 
d’autre {p. ex., un parent age), veuillez indiquer... 
rage du patient_ 

Le patient est,.. +t un homme ±2 une femme 


1 . Comment avez-vous entendu parier de fa 
Clinique pour la premiere fois? 

Mon medecin de familie me t’a recomrrtandee. 
Un autre professionnel de la sante me I’a 
recommand 6 e, 

Bulletin d’information ou journal cotnmunautarre 
J’ai repu de l‘informafion par la poste. 

Ami ou membre de la familie 
Affiche ou kiosque promotionnel 
Rencontre communautaire 
Publicite 
En ligne 

Autre (precisezj _ 


Depuis combien de temps frequentez-vous I 3 
Clinique? 

1-6 mois 
6-12 mots 
12-18 mois 
Plus de 18 mois 

Je ne sats pas, / Je ne me souviens pas. 

Environ combien de fois par annee frequentez- 
vous fa Clinique? 

Une fois par annee 

Tous les 6 mois 

Tous les 2 a 5 mois 

Une fois par mois 

Une fois par semaine 

Plus d’une fois par semaine 

Je ne sais pas. / Je ne me souviens pas. 


4. Quels types de soins avez-vous repus a la 
Clinique? (Cochez tout ce qui s'applique.) 

±61 Diagnostic ou traitement d'une maladie minaure 
{p. ex., rhume ou infection des oreilles) 

±02 Diagnostic ou traitement d’une blessure mineure 
(p. ex., entorse, coupure, eraflure) 

±03 Depistage d’une maladie chronique 
(p, ex., diabete) 

±04 Observation d’une maladie chronique stable 
(p ex., diabete, asthme, hypertension) 

±05 Soins durant la grossesse 

±06 Observation de la croissance et du 

developpement du nourrisson ou de I’enfant 

±07 Examen de sante annuel 
{p. ex , examen complet) 

±oa Education sur fa sante 

(p. ex,, concernant vos medicaments ou votre 
condition) 

±09 Counselling indMduel 

(p. ex., alimentation, cesser de fumer) 

±10 Orientation de groupe ou activites 6 ducatives 
{p. ex., cours prenataux, clinique sur la sante en 
mati^re de sexualite) 

±11 RPterence vers un sp 6 cialiste 
(p, ex., cardiologue) 

±12 Reference vers un autre professionnel de la sante 
(p. ex., physlotherapeute) 

±ss Autre (precisez):_ 


INSTRUCTIONS : Veuillez lire atfentivement chacune des questions et cochez (V) ou ecrivez votre reponse 
Veuillez refourner le questionnaire diiment rempli dans I’enveloppe preaffranchie ci-jointe. 
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5. Vous a-t-on deja explique le role de 
rinfirmier(ere) pratiden(ne) que vous consultez 
a !a Clinique? 

±01 Oui, un infirmier{Gre) praticien(ne) me fa explique 

±02 Our, un m^decin me Yb explique 

±03 Oui, une autre personne me I'a explique. 

(Prdcisez qui est cette 
personne): _ 

±oo Personne ne me Ta expfiqud. 

±88 Je ne sais pas / Je ne me souviens pas. 

6. Le role de rinfirmrer(ere) praticien(ne) est 
different de ceiui du medecrn de famille. 
Croyez-vous bien comprendre les differences 
qui existent? 

±01 Our 

±00 Non 

±S8 de ne sais pas. / Je suis incertain (e). 

7. Dans quelle mesure etes^vous satisfait{e) des 
services de votre infirmier(ere) praticien(ne)? 

±05 Tr^s satisfait(e) (Passez a la question 9.) 

±04 Plutot satfsfait(e) (Passez a la question 3,) 

±03 Neutre {ni satisfait(e) ni Insatisfaft(e)) 

(Passez a la question 9.) 

±02 PlutSt insatrsfait{e) (Passez a fa question 8,) 

±^t TrPs [nsatisfart(e) (Passez a la question 8.) 

8. Si vous etes insatisfait{e) des services que 
vous avez regus, veuillez en expliquer. 


10. Veuillez indiquer dans quelle mesure vous §tes 
d’accord ou non avec les enonc^s qui suivent. 
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Ilya toujours des sieges 
disponibles dans la salle 
d’attente.... .. 
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La saile d‘attente est 
oonfortable.,. 
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Les salles d’examen sont 
confortafales. 
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±3 

±2 

±1 

±s 


11. Veuillez indiquer dans quelle mesure vous etes 
d'accord ou non avec les enonc^s qui suivent. 
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Les residents de Sudbury 
ont un meilteur acc^s aux 
services de sante. 
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±4 
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9. Depuis que vous frequentez la Clinique, votre 
infirmier(ere) praticien (na) vous a-t-il(el!e) 
appris quelque chose sur votre etat de sante 
dont vous n'etiez pas au courant? 

±01 Oui (Veufflez pr^ciser fa condition) 


±00 Non 

±33 Je ne sais pas, / Je suis incertain(e). 


Le temps d'attente pour 
voir un (une) infirmier(^re) 
praticien(ne) est plus court 
que le temps J'attente pour 
voir un medecrn ne I'etait 
auparavant.... 

!l est plus facile d'etre 
ref ere a un autre 
professionne! de la sante 
(p. ex., un 

physiotherapeute) quand 
fen at besoin,... 

Je me sens mieux 
prepar^(e) pour prendre 
so in de mor-meme et gerer 
ma sante et mon bien-etre.. 


±5 ±4 


±5 ±4 


±5 ±4 


±3 


±3 


±3 


±2 


±2 


±1 ±8 


±1 ±a 


±1 ±3 


Je me sens mieux 
prepare(e) pour ptevenir fa 

maladie et eviter que mon ±s ±4 ±0 ±2 ±i ±e 

fetat de sant^ ne se 
deteriore.......... 


INSTRUCTIONS : Veuillez lire attentivement chacune des questions et cochez (^} ou ecrivez votre reponse. 
Vouillcz retourner Is questionnairs dument r©mpli dans I’enveloppe proaffranchi© ci-jointo. 
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12. Avez-vous dej^ partidpe aux programmes 
offerts par !a dinique (p. ex,, programme de 
cesser de turner, programme d’immunisation 
centre le VPH ou "Pourquoi se priver?") 

±01 Oui 

±00 Non (Passez ^ fa question 14.) 

±88 vie ne sais pas. / Je suis incertain(e). (Passez a 
la question 14.) 


14. De fagon gendrale, croyez-vous que les 
infirmier(ere)s praticien(ne}s constituent un 
moyen efficace d’ameliorer la prestation de 
soins de sante a Sudbury? 

±01 Oui 

±00 Non 

±88 Je ne sais pas. / Je suis mcertain(e). 


13. Veuillez indiquer votre degre de satisfaction 
envers les programmes auxquels vous avez 
partidpe. 
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Programme pour 
cesser de turner.. 

±5 

±4 

±3 

±2 

±t 

±0 

Programme 
d'immunisation 

+5 

±4 

±3 

±2 


±0 

contre le VPH. 






Programme 
"Pourquoi se 
priver?. 

±5 

±4 

±3 

±2 

±1 

±0 


15. Veuiliez expliquer pourquoi ou pourquoi pas 





Nous vous remercions infiniment de votre collaboration. 

Veuillez retourner le questionnaire dument rempli d'ici le 17 Juillet 2009 dans I’enveloppe 

preaffranchie ci-jointe a I’adresse qui suit: 

PRA Inc. 

500-363 Broadway 
Winnipeg, Manitoba R3C 3N9 




JNSTRUCTIONS : Veuillez lire attenlivement chacune des questions et cochez (^) ou ecrivez votre reponse 
Veutfiez retourner le questionnaire dument rempli dans I’enveloppe preaffranchie ci-jointe. 




































APPENDIX E 

Reminder postcards for Patient Feedback Survey 



Ministry of Health and Long-Term Care 

Evaluation of the Sudbury District Nurse Practitioner Clinics—October I, 2009 



Patient Feedback Survey Sondage aupres des patients 

Recently, the Sudbury District Nurse Practitioner Clinics sent you a questionnaire. If you 
have already completed and returned it, please accept our sincere thanks. If not, please do 
so at your earliest convenience, by July 17, 2009 at the latest. 

The survey is directed to only a small group of individuals Your feedback is very important to 
us. If you require anofrier copy of the questionnaire, please call Sarah F-aser (toll free) at 
1-866-422^8468. 


R^emment, la clinIque Sudbury District Nurse Practitioner vous a expedie un 
questionnaire. Si vous I'avez deja complete et retourne, nous vous en remercions. Sinon, 
nous vous prions des que possible, au plus tard le 17 juillet 2009. 

Le sondage est distribu6 aupres d'un petit groupe d'individus. Votre retroaction est tres 
importante a nos yeux. Si vous avez besom d'une autre copie du questionnaire, veuillez 
communiquer avec Sarah Fraser (sans frais) au 1-866-422-8468. 

Marilyn Butcher 

Clinic Director, Sudbury District Nurse Practitioner Clinics 
Directrice, Sudbury District Nurse Practitioner Clinics 







